Sep 10 16:15:26.502252 ip-10-0-139-24 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 16:15:26.502266 ip-10-0-139-24 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 16:15:26.502277 ip-10-0-139-24 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 16:15:26.502875 ip-10-0-139-24 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 16:15:38.091479 ip-10-0-139-24 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 16:15:38.091497 ip-10-0-139-24 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e10be8d0835048bb93c2388ea58289fa -- Sep 10 16:17:29.152350 ip-10-0-139-24 systemd[1]: Starting Kubernetes Kubelet... Sep 10 16:17:29.562686 ip-10-0-139-24 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 16:17:29.562686 ip-10-0-139-24 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 16:17:29.562686 ip-10-0-139-24 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 16:17:29.562686 ip-10-0-139-24 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 16:17:29.562686 ip-10-0-139-24 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 16:17:29.565481 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.565389 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 16:17:29.570587 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570568 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:17:29.570587 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570587 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:17:29.570587 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570590 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570593 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570597 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570600 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570603 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570606 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570609 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570612 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570614 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570617 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570620 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570622 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570625 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570629 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570631 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570634 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570636 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570639 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570642 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570646 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:17:29.570683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570649 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570652 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570654 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570657 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570660 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570662 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570665 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570668 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570671 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570674 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570677 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570679 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570682 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570685 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570688 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570692 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570694 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570698 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570701 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:17:29.571235 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570704 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570707 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570709 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570712 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570715 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570718 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570720 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570724 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570729 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570731 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570734 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570737 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570739 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570742 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570744 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570748 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570750 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570753 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570756 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570759 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:17:29.571699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570761 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570764 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570769 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570773 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570776 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570779 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570782 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570785 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570787 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570790 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570793 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570796 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570799 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570802 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570804 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570807 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570810 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570813 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570815 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570818 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:17:29.572215 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570820 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570823 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570826 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570828 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570832 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570835 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570838 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570842 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570845 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570848 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570851 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570854 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570856 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570859 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570862 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570864 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570868 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.570870 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572617 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:17:29.572716 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572625 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572628 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572631 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572635 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572639 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572642 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572645 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572647 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572650 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572653 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572655 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572658 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572661 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572663 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572666 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572668 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572671 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572673 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572676 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572679 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:17:29.573318 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572682 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572685 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572687 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572690 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572693 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572695 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572698 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572701 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572703 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572707 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572710 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572712 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572715 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572717 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572720 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572723 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572726 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572730 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572732 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572735 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:17:29.573873 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572737 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572740 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572743 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572745 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572748 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572750 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572753 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572755 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572758 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572760 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572763 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572766 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572770 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572773 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572775 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572778 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572781 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572783 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572786 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572789 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:17:29.574404 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572792 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572795 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572798 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572800 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572805 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572809 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572812 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572815 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572818 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572822 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572826 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572829 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572832 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572834 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572837 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572839 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572842 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572844 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572847 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:17:29.574950 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572850 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572852 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572855 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572857 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572860 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572863 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572866 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572870 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572874 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572877 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572880 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572882 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572885 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572888 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572891 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572893 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572896 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572899 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.572902 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:17:29.575461 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573344 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573351 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573353 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573356 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573361 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573364 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573366 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573369 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573373 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573377 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573380 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573383 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573386 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573389 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573392 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573394 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573397 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573401 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573405 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:17:29.575942 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573408 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573411 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573414 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573417 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573420 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573424 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573427 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573430 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573433 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573436 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573440 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573442 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573445 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573448 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573450 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573453 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573456 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573458 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573462 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573464 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:17:29.576445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573467 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573469 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573472 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573474 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573477 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573479 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573482 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573485 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573487 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573490 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573492 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573495 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573497 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573500 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573502 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573505 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573508 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573510 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573513 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573516 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:17:29.576933 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573518 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573521 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573523 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573526 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573528 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573531 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573533 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573536 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573539 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573542 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573544 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573552 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573555 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573558 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573560 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573563 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573565 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573568 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573571 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573574 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:17:29.577433 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573576 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573579 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573582 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573588 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573590 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573593 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573596 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573598 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573601 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573603 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573606 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573608 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573611 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573613 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573616 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573618 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573621 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573624 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573626 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.573629 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:17:29.577926 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573705 2562 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573742 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573751 2562 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573756 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573769 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573773 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573782 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573786 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573790 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573793 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573797 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573800 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573804 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573807 2562 flags.go:64] FLAG: --cgroup-root="" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573810 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573815 2562 flags.go:64] FLAG: --client-ca-file="" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573818 2562 flags.go:64] FLAG: --cloud-provider="external" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573821 2562 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573828 2562 flags.go:64] FLAG: --cluster-domain="" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573832 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573835 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573839 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573843 2562 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573847 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 16:17:29.578423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573850 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573853 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573857 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573860 2562 flags.go:64] FLAG: --contention-profiling="false" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573863 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573866 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573869 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573872 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573876 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573879 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573883 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573886 2562 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573889 2562 flags.go:64] FLAG: --enable-server="true" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573893 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573897 2562 flags.go:64] FLAG: --event-burst="100" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573900 2562 flags.go:64] FLAG: --event-qps="50" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573903 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573906 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573909 2562 flags.go:64] FLAG: --eviction-hard="" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573913 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573916 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573919 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573922 2562 flags.go:64] FLAG: --eviction-soft="" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573925 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573928 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 16:17:29.579068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573931 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573934 2562 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573937 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573940 2562 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573943 2562 flags.go:64] FLAG: --feature-gates="" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573951 2562 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573954 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573957 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573960 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573963 2562 flags.go:64] FLAG: --healthz-port="10248" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573980 2562 flags.go:64] FLAG: --help="false" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573984 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573987 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573991 2562 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573994 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.573997 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574001 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574004 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574006 2562 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574009 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574012 2562 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574017 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574020 2562 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574023 2562 flags.go:64] FLAG: --kube-reserved="" Sep 10 16:17:29.579696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574026 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574029 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574032 2562 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574051 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574056 2562 flags.go:64] FLAG: --lock-file="" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574060 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574065 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574068 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574074 2562 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574077 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574080 2562 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574083 2562 flags.go:64] FLAG: --logging-format="text" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574086 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574089 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574092 2562 flags.go:64] FLAG: --manifest-url="" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574095 2562 flags.go:64] FLAG: --manifest-url-header="" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574100 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574103 2562 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574107 2562 flags.go:64] FLAG: --max-pods="110" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574110 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574114 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574116 2562 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574120 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574122 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574125 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 16:17:29.580285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574128 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574136 2562 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574139 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574142 2562 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574145 2562 flags.go:64] FLAG: --pod-cidr="" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574149 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574155 2562 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574158 2562 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574161 2562 flags.go:64] FLAG: --pods-per-core="0" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574165 2562 flags.go:64] FLAG: --port="10250" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574168 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574171 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08556ccc24c5c01f1" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574175 2562 flags.go:64] FLAG: --qos-reserved="" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574178 2562 flags.go:64] FLAG: --read-only-port="10255" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574181 2562 flags.go:64] FLAG: --register-node="true" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574184 2562 flags.go:64] FLAG: --register-with-taints="" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574188 2562 flags.go:64] FLAG: --registry-burst="10" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574191 2562 flags.go:64] FLAG: --registry-qps="5" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574194 2562 flags.go:64] FLAG: --reserved-cpus="" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574197 2562 flags.go:64] FLAG: --reserved-memory="" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574201 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574204 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574207 2562 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574210 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574213 2562 flags.go:64] FLAG: --runonce="false" Sep 10 16:17:29.580958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574216 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574219 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574222 2562 flags.go:64] FLAG: --seccomp-default="false" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574225 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574228 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574231 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574234 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574238 2562 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574240 2562 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574243 2562 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574246 2562 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574249 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574260 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574265 2562 flags.go:64] FLAG: --system-cgroups="" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574268 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574273 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574278 2562 flags.go:64] FLAG: --tls-cert-file="" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574282 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574287 2562 flags.go:64] FLAG: --tls-min-version="" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574290 2562 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574292 2562 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574295 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574298 2562 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574301 2562 flags.go:64] FLAG: --v="2" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574306 2562 flags.go:64] FLAG: --version="false" Sep 10 16:17:29.581598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574310 2562 flags.go:64] FLAG: --vmodule="" Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574314 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.574318 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574443 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574448 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574451 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574454 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574457 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574460 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574463 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574466 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574469 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574472 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574474 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574477 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574479 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574483 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574487 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574491 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:17:29.582227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574495 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574498 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574507 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574510 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574515 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574518 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574521 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574524 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574526 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574529 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574531 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574534 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574537 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574539 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574542 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574545 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574547 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574550 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574553 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574555 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:17:29.582699 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574558 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574561 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574563 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574566 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574568 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574571 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574573 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574576 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574579 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574581 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574584 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574586 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574589 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574591 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574594 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574597 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574601 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574605 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574607 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574610 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:17:29.583217 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574613 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574616 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574620 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574623 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574626 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574629 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574632 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574635 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574638 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574640 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574643 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574646 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574648 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574651 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574654 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574656 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574659 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574662 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574664 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:17:29.583708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574667 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574669 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574672 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574675 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574677 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574680 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574682 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574685 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574687 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574692 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574695 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574698 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574700 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574703 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574706 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574709 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574711 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574714 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574717 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574719 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:17:29.584225 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574722 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574724 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574727 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.574729 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.575323 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.582443 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.582461 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.583168 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.583176 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583241 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583250 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583253 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583256 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583260 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:17:29.584720 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583263 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583266 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583269 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583272 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583275 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583278 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583281 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583285 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583287 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583290 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583293 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583296 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583299 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583301 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583304 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583307 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583310 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583313 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583316 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583318 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:17:29.585125 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583321 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583323 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583326 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583329 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583332 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583335 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583338 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583342 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583346 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583349 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583352 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583355 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583358 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583361 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583363 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583366 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583368 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583371 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583373 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:17:29.585615 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583376 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583379 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583382 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583385 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583387 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583390 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583392 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583395 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583398 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583400 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583403 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583405 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583408 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583411 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583415 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583418 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583421 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583424 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583427 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583430 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:17:29.586092 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583432 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583434 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583437 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583439 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583442 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583445 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583447 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583450 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583452 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583455 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583458 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583460 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583463 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583465 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583469 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583471 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583474 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583476 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583479 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583481 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:17:29.586590 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583485 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583487 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583490 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583492 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583496 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583498 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583501 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583503 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583506 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583509 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583512 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583515 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583517 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583520 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:17:29.587087 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583522 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.583528 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583676 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583681 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583684 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583687 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583690 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583694 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583698 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583701 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583704 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583707 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583710 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583713 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583716 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 16:17:29.587469 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583719 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583722 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583724 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583727 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583730 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583733 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583736 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583739 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583743 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583746 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583749 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583751 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583754 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583757 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583760 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583762 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583765 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583767 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583770 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 16:17:29.587854 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583773 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583775 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583778 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583780 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583783 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583785 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583788 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583790 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583793 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583795 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583798 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583800 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583804 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583806 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583809 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583811 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583814 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583816 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583819 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583822 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 16:17:29.588338 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583825 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583827 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583830 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583832 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583835 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583838 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583841 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583845 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583848 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583851 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583854 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583857 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583860 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583862 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583865 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583868 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583870 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583873 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583875 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583878 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 16:17:29.588838 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583880 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583883 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583885 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583888 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583890 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583893 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583896 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583899 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583901 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583904 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583906 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583909 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583912 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583914 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583917 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583919 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583922 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583925 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583927 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583930 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 16:17:29.589501 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583932 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583935 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583938 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583940 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583943 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583945 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:29.583948 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.583953 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.584108 2562 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.586085 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.586974 2562 server.go:1019] "Starting client certificate rotation" Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.587080 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.587120 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 16:17:29.590004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.589415 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 16:17:29.591881 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.591742 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 16:17:29.602337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.602313 2562 log.go:25] "Validated CRI v1 runtime API" Sep 10 16:17:29.605021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.605005 2562 log.go:25] "Validated CRI v1 image API" Sep 10 16:17:29.605942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.605925 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 16:17:29.608503 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.608485 2562 fs.go:135] Filesystem UUIDs: map[2a888a39-e075-4dcf-a541-a31101b7e9f7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 df0dc279-4f70-4c8d-907b-c1992db1426d:/dev/nvme0n1p3] Sep 10 16:17:29.608551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.608503 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 16:17:29.614606 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.614484 2562 manager.go:216] Machine: {Timestamp:2026-09-10 16:17:29.612991531 +0000 UTC m=+0.339310453 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099388 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ca45c094f9ba504a3e14a2129064 SystemUUID:ec21ca45-c094-f9ba-504a-3e14a2129064 BootID:e10be8d0-8350-48bb-93c2-388ea58289fa Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:de:e0:61:91:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:de:e0:61:91:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:3d:38:da:01:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 16:17:29.614606 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.614590 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 16:17:29.614747 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.614684 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 16:17:29.615321 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.615293 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 16:17:29.618975 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.618946 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 16:17:29.619172 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.618981 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-24.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 16:17:29.619850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.619838 2562 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 16:17:29.619906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.619858 2562 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 16:17:29.619906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.619869 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 16:17:29.620489 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.620477 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 16:17:29.621860 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.621848 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 16:17:29.624948 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.624933 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 10 16:17:29.625114 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.625103 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 16:17:29.625186 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.625144 2562 kubelet.go:486] "Attempting to sync node with API server" Sep 10 16:17:29.625186 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.625168 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 16:17:29.625274 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.625190 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 16:17:29.625274 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.625205 2562 kubelet.go:398] "Adding apiserver pod source" Sep 10 16:17:29.625274 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.625218 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 16:17:29.625618 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.625603 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 16:17:29.625679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.625629 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 16:17:29.628130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.628111 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 16:17:29.630506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.630488 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 16:17:29.630573 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.630522 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 16:17:29.630723 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.630714 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 16:17:29.630760 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.630728 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 16:17:29.630760 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.630735 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 16:17:29.630760 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.630740 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 16:17:29.630760 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.630748 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 16:17:29.630760 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.630754 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 16:17:29.631130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.631122 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 16:17:29.631130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.631130 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 16:17:29.631194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.631137 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 16:17:29.631194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.631143 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 16:17:29.631194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.631153 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 16:17:29.631194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.631161 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 16:17:29.632732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.632721 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 16:17:29.632767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.632733 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 16:17:29.636363 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.636247 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zzsjt" Sep 10 16:17:29.637097 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.637076 2562 server.go:1268] "Started kubelet" Sep 10 16:17:29.638395 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.638338 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 16:17:29.638402 ip-10-0-139-24 systemd[1]: Started Kubernetes Kubelet. Sep 10 16:17:29.638814 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.638417 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 16:17:29.639663 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.639644 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 16:17:29.640962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.640937 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 16:17:29.641055 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.641015 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-24.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 16:17:29.641643 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.641611 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-24.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 16:17:29.641643 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.641631 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 16:17:29.645145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.645126 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zzsjt" Sep 10 16:17:29.645759 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.645739 2562 server.go:310] "Adding debug handlers to kubelet server" Sep 10 16:17:29.646083 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.646062 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 16:17:29.648101 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.647104 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-24.ec2.internal.18d401e866824dd1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-24.ec2.internal,UID:ip-10-0-139-24.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-24.ec2.internal,},FirstTimestamp:2026-09-10 16:17:29.636785617 +0000 UTC m=+0.363104538,LastTimestamp:2026-09-10 16:17:29.636785617 +0000 UTC m=+0.363104538,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-24.ec2.internal,}" Sep 10 16:17:29.648264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.648242 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 16:17:29.648695 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.648679 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 16:17:29.648863 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.648838 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:29.649344 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649327 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 16:17:29.649449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649348 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 16:17:29.649510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649463 2562 reconstruct.go:97] "Volume reconstruction finished" Sep 10 16:17:29.649510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649470 2562 reconciler.go:29] "Reconciler: start to sync state" Sep 10 16:17:29.649510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649493 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 16:17:29.649722 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649707 2562 factory.go:55] Registering systemd factory Sep 10 16:17:29.649759 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649728 2562 factory.go:223] Registration of the systemd container factory successfully Sep 10 16:17:29.649965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649951 2562 factory.go:153] Registering CRI-O factory Sep 10 16:17:29.650004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.649969 2562 factory.go:223] Registration of the crio container factory successfully Sep 10 16:17:29.650057 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.650015 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 16:17:29.650057 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.650055 2562 factory.go:103] Registering Raw factory Sep 10 16:17:29.650123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.650068 2562 manager.go:1196] Started watching for new ooms in manager Sep 10 16:17:29.650540 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.650528 2562 manager.go:349] Starting recovery of all containers Sep 10 16:17:29.658988 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.658962 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:17:29.661846 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.661823 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-24.ec2.internal\" not found" node="ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.662654 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.662641 2562 manager.go:354] Recovery completed Sep 10 16:17:29.664407 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.664385 2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 10 16:17:29.669455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.669432 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:17:29.671363 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.671343 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:17:29.671439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.671378 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:17:29.671439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.671394 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:17:29.672020 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.672003 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 16:17:29.672020 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.672018 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 16:17:29.672195 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.672024 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:17:29.672195 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.672049 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 10 16:17:29.672749 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.672730 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:17:29.672828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.672762 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:17:29.672828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.672777 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:17:29.673448 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.673433 2562 policy_none.go:49] "None policy: Start" Sep 10 16:17:29.673500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.673452 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 16:17:29.673500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.673461 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 16:17:29.674350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.674338 2562 policy_none.go:47] "Start" Sep 10 16:17:29.749086 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.748907 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:29.750141 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.749252 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 16:17:29.750482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.750464 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 16:17:29.750542 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.750528 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 16:17:29.750577 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.750563 2562 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 16:17:29.750701 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.750665 2562 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 10 16:17:29.753261 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.753241 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:17:29.840629 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.840606 2562 manager.go:337] "Starting Device Plugin manager" Sep 10 16:17:29.840732 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.840658 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 16:17:29.840732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.840671 2562 server.go:87] "Starting device plugin registration server" Sep 10 16:17:29.841070 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.841054 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 16:17:29.841147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.841073 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 16:17:29.841197 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.841173 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 16:17:29.841335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.841263 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 16:17:29.841335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.841282 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 16:17:29.842031 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.842010 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 16:17:29.842123 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.842084 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:29.851488 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.851469 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal"] Sep 10 16:17:29.851576 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.851528 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:17:29.852426 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.852410 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:17:29.852523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.852446 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:17:29.852523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.852460 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:17:29.853877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.853862 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:17:29.853981 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.853964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.854030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.854007 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:17:29.854586 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.854572 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:17:29.854651 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.854600 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:17:29.854651 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.854612 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:17:29.855098 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.855083 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:17:29.855153 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.855114 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:17:29.855153 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.855129 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:17:29.856305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.856292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.856357 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.856319 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:17:29.856990 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.856973 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:17:29.857066 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.857002 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:17:29.857066 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.857014 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:17:29.878940 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.878910 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-24.ec2.internal\" not found" node="ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.883360 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.883341 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-24.ec2.internal\" not found" node="ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.941962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.941929 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 16:17:29.942957 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.942938 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientMemory" Sep 10 16:17:29.943069 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.942971 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 16:17:29.943069 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.942981 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeHasSufficientPID" Sep 10 16:17:29.943069 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.943005 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.950597 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.950573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e031a7be8eb499dcf51c16d6ce862b92-config\") pod \"kube-apiserver-proxy-ip-10-0-139-24.ec2.internal\" (UID: \"e031a7be8eb499dcf51c16d6ce862b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.950677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.950602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5f367cb344977acd80fcfe8020385ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal\" (UID: \"a5f367cb344977acd80fcfe8020385ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.950677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.950621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5f367cb344977acd80fcfe8020385ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal\" (UID: \"a5f367cb344977acd80fcfe8020385ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.951529 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:29.951514 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-24.ec2.internal" Sep 10 16:17:29.951570 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.951538 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-24.ec2.internal\": node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:29.966353 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:29.966325 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:30.051140 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.051108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e031a7be8eb499dcf51c16d6ce862b92-config\") pod \"kube-apiserver-proxy-ip-10-0-139-24.ec2.internal\" (UID: \"e031a7be8eb499dcf51c16d6ce862b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.051282 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.051148 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5f367cb344977acd80fcfe8020385ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal\" (UID: \"a5f367cb344977acd80fcfe8020385ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.051282 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.051175 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5f367cb344977acd80fcfe8020385ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal\" (UID: \"a5f367cb344977acd80fcfe8020385ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.051282 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.051235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5f367cb344977acd80fcfe8020385ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal\" (UID: \"a5f367cb344977acd80fcfe8020385ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.051282 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.051236 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5f367cb344977acd80fcfe8020385ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal\" (UID: \"a5f367cb344977acd80fcfe8020385ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.051282 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.051248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e031a7be8eb499dcf51c16d6ce862b92-config\") pod \"kube-apiserver-proxy-ip-10-0-139-24.ec2.internal\" (UID: \"e031a7be8eb499dcf51c16d6ce862b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.067254 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:30.067227 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:30.168178 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:30.168083 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:30.182306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.182280 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.186353 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.186227 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.268886 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:30.268833 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:30.369519 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:30.369473 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:30.470062 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:30.469953 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:30.571037 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:30.571006 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:30.587396 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.587366 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 16:17:30.587588 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.587560 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 16:17:30.587647 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.587576 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 16:17:30.649075 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.649009 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 16:12:29 +0000 UTC" deadline="2028-03-31 05:53:12.183167478 +0000 UTC" Sep 10 16:17:30.649075 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.649067 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13621h35m41.534105352s" Sep 10 16:17:30.671198 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:30.671163 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-24.ec2.internal\" not found" Sep 10 16:17:30.755155 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:30.755099 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f367cb344977acd80fcfe8020385ad.slice/crio-61f0050e991d5f65129fde19d6140de007ef01c6fd85a4e6d494c71fc6c5fecf WatchSource:0}: Error finding container 61f0050e991d5f65129fde19d6140de007ef01c6fd85a4e6d494c71fc6c5fecf: Status 404 returned error can't find the container with id 61f0050e991d5f65129fde19d6140de007ef01c6fd85a4e6d494c71fc6c5fecf Sep 10 16:17:30.755696 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:30.755673 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode031a7be8eb499dcf51c16d6ce862b92.slice/crio-eb2238a8d55b733e619f97a3db9a37298d1b682ae9efb6e48e3690fbb204256a WatchSource:0}: Error finding container eb2238a8d55b733e619f97a3db9a37298d1b682ae9efb6e48e3690fbb204256a: Status 404 returned error can't find the container with id eb2238a8d55b733e619f97a3db9a37298d1b682ae9efb6e48e3690fbb204256a Sep 10 16:17:30.759677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.759663 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:17:30.766916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.766897 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:17:30.849564 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.849519 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.861112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.861083 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 16:17:30.863181 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.863163 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" Sep 10 16:17:30.873972 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:30.873945 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 16:17:31.084833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.084744 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:17:31.481932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.481845 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:17:31.627303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.627248 2562 apiserver.go:52] "Watching apiserver" Sep 10 16:17:31.635725 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.635701 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 16:17:31.636149 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.636125 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8","openshift-cluster-node-tuning-operator/tuned-4c984","openshift-image-registry/node-ca-kcxpt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal","kube-system/global-pull-secret-syncer-l849b","openshift-dns/node-resolver-8jv76","openshift-multus/multus-additional-cni-plugins-jvhqn","openshift-multus/multus-ppcww","openshift-multus/network-metrics-daemon-hn92b","openshift-network-diagnostics/network-check-target-mvhbl","openshift-network-operator/iptables-alerter-9gnd7","openshift-ovn-kubernetes/ovnkube-node-h2jpq","kube-system/konnectivity-agent-4544q","kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal"] Sep 10 16:17:31.637786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.637763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:31.637894 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.637867 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:31.638874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.638857 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:31.640070 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.640052 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.641452 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.641431 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.642020 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.642003 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4vz7z\"" Sep 10 16:17:31.642458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.642405 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 16:17:31.642554 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.642534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 16:17:31.642787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.642767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.642875 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.642849 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:31.642942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.642878 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 16:17:31.642942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.642919 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 16:17:31.643151 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.643130 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 16:17:31.644161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.644138 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 16:17:31.644553 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.644534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r7t2d\"" Sep 10 16:17:31.644821 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.644770 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.644821 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.644792 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 16:17:31.644960 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.644834 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p25v8\"" Sep 10 16:17:31.645268 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.644779 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 16:17:31.645570 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.645415 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 16:17:31.645570 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.645470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:31.645570 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.645531 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:31.646698 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.646676 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.647997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.647981 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.649160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.649144 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 16:17:31.649261 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.649246 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.649561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.649547 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 16:17:31.650213 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.650194 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 16:17:31.650317 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.650214 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pbklj\"" Sep 10 16:17:31.650317 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.650310 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 16:17:31.650666 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.650649 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sqrnb\"" Sep 10 16:17:31.650751 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.650685 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 16:17:31.650953 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.650936 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.652362 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.651833 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:17:31.652362 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.651936 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 16:17:31.652362 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.652128 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gqgs9\"" Sep 10 16:17:31.652965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.652781 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 16:17:31.653058 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.653015 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8llsm\"" Sep 10 16:17:31.653594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.653153 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 16:17:31.655368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.654023 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 16:17:31.655368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.654572 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 16:17:31.655368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.654680 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hmg75\"" Sep 10 16:17:31.655368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.655267 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:17:31.656147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.656126 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.656873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.656855 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 16:17:31.659464 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.659446 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 16:17:31.659996 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.659972 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 16:17:31.660104 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.660006 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c5t4r\"" Sep 10 16:17:31.661327 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.661311 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 16:17:31.661406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.661340 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 16:17:31.661406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.661353 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 16:17:31.661522 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.661445 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 16:17:31.662419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-kubelet\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.662518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662432 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-hostroot\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.662518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662457 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-kubelet-config\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.662518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-modprobe-d\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.662669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-cni-bin\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.662669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-daemon-config\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.662669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.662669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662604 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f12a9ba-2874-4c63-803e-b40f109453ac-host\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.662669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.662669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysctl-d\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.662906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-system-cni-dir\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.662906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8q6z\" (UniqueName: \"kubernetes.io/projected/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-kube-api-access-x8q6z\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.662906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-host\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.662906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662745 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-dbus\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.662906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662771 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-run\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.662906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.662906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-kubernetes\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.662906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkrr\" (UniqueName: \"kubernetes.io/projected/0f12a9ba-2874-4c63-803e-b40f109453ac-kube-api-access-qbkrr\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l748\" (UniqueName: \"kubernetes.io/projected/944b5648-71ef-4980-927f-351334db589e-kube-api-access-7l748\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndln\" (UniqueName: \"kubernetes.io/projected/30528697-a751-4fec-9d6b-d7446534904e-kube-api-access-wndln\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.662998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30528697-a751-4fec-9d6b-d7446534904e-tmp\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-iptables-alerter-script\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-cni-multus\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-conf-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-device-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxzl\" (UniqueName: \"kubernetes.io/projected/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-kube-api-access-wsxzl\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-sys\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:31.663325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-netns\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-tmp-dir\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysconfig\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-var-lib-kubelet\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzvf\" (UniqueName: \"kubernetes.io/projected/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-kube-api-access-2pzvf\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a809f2d7-03d7-487d-a48e-4628fc304d2a-agent-certs\") pod \"konnectivity-agent-4544q\" (UID: \"a809f2d7-03d7-487d-a48e-4628fc304d2a\") " pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663738 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysctl-conf\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30528697-a751-4fec-9d6b-d7446534904e-etc-tuned\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.663946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-cni-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663927 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-hosts-file\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.663979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-cnibin\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-cni-binary-copy\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-socket-dir-parent\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-k8s-cni-cncf-io\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-multus-certs\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-etc-kubernetes\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664347 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqg4q\" (UniqueName: \"kubernetes.io/projected/ad37714e-ec26-404b-8b83-76e78c1d12a5-kube-api-access-tqg4q\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.664417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f12a9ba-2874-4c63-803e-b40f109453ac-serviceca\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-lib-modules\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664454 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a809f2d7-03d7-487d-a48e-4628fc304d2a-konnectivity-ca\") pod \"konnectivity-agent-4544q\" (UID: \"a809f2d7-03d7-487d-a48e-4628fc304d2a\") " pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-system-cni-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prld\" (UniqueName: \"kubernetes.io/projected/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-kube-api-access-7prld\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664624 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-systemd\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664640 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-host-slash\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.664940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.664654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-cnibin\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.666688 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.666661 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 16:17:31.693023 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.692980 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dp6xx" Sep 10 16:17:31.702365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.702325 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dp6xx" Sep 10 16:17:31.751106 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.751006 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 16:17:31.756400 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.756344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" event={"ID":"a5f367cb344977acd80fcfe8020385ad","Type":"ContainerStarted","Data":"61f0050e991d5f65129fde19d6140de007ef01c6fd85a4e6d494c71fc6c5fecf"} Sep 10 16:17:31.758850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.758820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" event={"ID":"e031a7be8eb499dcf51c16d6ce862b92","Type":"ContainerStarted","Data":"eb2238a8d55b733e619f97a3db9a37298d1b682ae9efb6e48e3690fbb204256a"} Sep 10 16:17:31.765491 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxzl\" (UniqueName: \"kubernetes.io/projected/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-kube-api-access-wsxzl\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.765611 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-sys\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.765611 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:31.765611 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-netns\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.765611 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765592 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-tmp-dir\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.765837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765623 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-cni-bin\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.765837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysconfig\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.765837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-var-lib-kubelet\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.765837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzvf\" (UniqueName: \"kubernetes.io/projected/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-kube-api-access-2pzvf\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.765837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a809f2d7-03d7-487d-a48e-4628fc304d2a-agent-certs\") pod \"konnectivity-agent-4544q\" (UID: \"a809f2d7-03d7-487d-a48e-4628fc304d2a\") " pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:31.765837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysctl-conf\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.765837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:31.765837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30528697-a751-4fec-9d6b-d7446534904e-etc-tuned\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-cni-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-hosts-file\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.765958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-cnibin\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-cni-binary-copy\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-socket-dir-parent\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.766154 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-k8s-cni-cncf-io\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-multus-certs\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-etc-kubernetes\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.766227 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs podName:944b5648-71ef-4980-927f-351334db589e nodeName:}" failed. No retries permitted until 2026-09-10 16:17:32.266203274 +0000 UTC m=+2.992522205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs") pod "network-metrics-daemon-hn92b" (UID: "944b5648-71ef-4980-927f-351334db589e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766250 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-etc-kubernetes\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766264 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqg4q\" (UniqueName: \"kubernetes.io/projected/ad37714e-ec26-404b-8b83-76e78c1d12a5-kube-api-access-tqg4q\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.766296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-sys\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-ovn\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f12a9ba-2874-4c63-803e-b40f109453ac-serviceca\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766376 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-netns\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-lib-modules\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a809f2d7-03d7-487d-a48e-4628fc304d2a-konnectivity-ca\") pod \"konnectivity-agent-4544q\" (UID: \"a809f2d7-03d7-487d-a48e-4628fc304d2a\") " pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-system-cni-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-kubelet\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prld\" (UniqueName: \"kubernetes.io/projected/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-kube-api-access-7prld\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-node-log\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/59c257de-f095-4753-adb3-14789e3b440d-ovn-node-metrics-cert\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-systemd\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766664 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-var-lib-kubelet\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-host-slash\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766714 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-tmp-dir\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.767056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-cnibin\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-log-socket\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766787 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-env-overrides\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-kubelet\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-hostroot\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-kubelet-config\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766884 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysctl-conf\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-run-netns\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766914 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-systemd\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-cni-netd\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766937 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-registration-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbnk\" (UniqueName: \"kubernetes.io/projected/59c257de-f095-4753-adb3-14789e3b440d-kube-api-access-7gbnk\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-modprobe-d\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-systemd\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-cni-bin\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-daemon-config\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.767993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767077 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f12a9ba-2874-4c63-803e-b40f109453ac-host\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767215 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-host-slash\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-cni-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-hosts-file\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767279 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-cnibin\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767312 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-cnibin\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767354 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-kubelet\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767361 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767386 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-hostroot\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767421 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-kubelet-config\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767554 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f12a9ba-2874-4c63-803e-b40f109453ac-serviceca\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-ovnkube-config\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767674 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-system-cni-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-lib-modules\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.768934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767872 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysctl-d\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767725 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysctl-d\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-system-cni-dir\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.767973 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8q6z\" (UniqueName: \"kubernetes.io/projected/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-kube-api-access-x8q6z\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.766633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-sysconfig\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-run-ovn-kubernetes\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-host\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-var-lib-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768153 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-modprobe-d\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-dbus\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768156 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-run\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768212 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-sys-fs\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-systemd-units\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-cni-bin\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-ovnkube-script-lib\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.769880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768327 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-run\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768327 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-k8s-cni-cncf-io\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768340 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a809f2d7-03d7-487d-a48e-4628fc304d2a-konnectivity-ca\") pod \"konnectivity-agent-4544q\" (UID: \"a809f2d7-03d7-487d-a48e-4628fc304d2a\") " pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-run-multus-certs\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768415 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-dbus\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768414 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-system-cni-dir\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768517 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-host\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-socket-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-socket-dir-parent\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-kubernetes\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkrr\" (UniqueName: \"kubernetes.io/projected/0f12a9ba-2874-4c63-803e-b40f109453ac-kube-api-access-qbkrr\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f12a9ba-2874-4c63-803e-b40f109453ac-host\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l748\" (UniqueName: \"kubernetes.io/projected/944b5648-71ef-4980-927f-351334db589e-kube-api-access-7l748\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768695 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndln\" (UniqueName: \"kubernetes.io/projected/30528697-a751-4fec-9d6b-d7446534904e-kube-api-access-wndln\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768728 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.770815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-slash\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.771689 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30528697-a751-4fec-9d6b-d7446534904e-tmp\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.771689 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768869 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-daemon-config\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.771689 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-iptables-alerter-script\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.771689 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.768904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-cni-multus\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.775498 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-cni-binary-copy\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.775558 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-conf-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.775607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-device-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.775648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-etc-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.775795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-device-dir\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.775961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30528697-a751-4fec-9d6b-d7446534904e-etc-kubernetes\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.776022 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.776202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad37714e-ec26-404b-8b83-76e78c1d12a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.777103 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.777086 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:31.777748 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.777181 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret podName:3cb5f5c7-9cca-44c3-9013-6fcc3788f877 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:32.277153257 +0000 UTC m=+3.003472166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret") pod "global-pull-secret-syncer-l849b" (UID: "3cb5f5c7-9cca-44c3-9013-6fcc3788f877") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:31.777748 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.777708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad37714e-ec26-404b-8b83-76e78c1d12a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.777871 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.777729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-iptables-alerter-script\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.777871 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.777790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-multus-conf-dir\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.777966 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.777871 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-host-var-lib-cni-multus\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.781407 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.781286 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30528697-a751-4fec-9d6b-d7446534904e-etc-tuned\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.781864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.781735 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a809f2d7-03d7-487d-a48e-4628fc304d2a-agent-certs\") pod \"konnectivity-agent-4544q\" (UID: \"a809f2d7-03d7-487d-a48e-4628fc304d2a\") " pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:31.784427 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.784373 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30528697-a751-4fec-9d6b-d7446534904e-tmp\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.793059 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.793027 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxzl\" (UniqueName: \"kubernetes.io/projected/dff7f8c0-5a53-45bc-8b61-8abebb4bf30c-kube-api-access-wsxzl\") pod \"node-resolver-8jv76\" (UID: \"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c\") " pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.801254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.801227 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 16:17:31.801932 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.801911 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:17:31.802043 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.801937 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:17:31.802043 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.801948 2562 projected.go:196] Error preparing data for projected volume kube-api-access-98xwl for pod openshift-network-diagnostics/network-check-target-mvhbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:31.802043 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:31.802001 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl podName:ed0d041c-4673-444b-8392-02de7cc0958f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:32.301985929 +0000 UTC m=+3.028304837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-98xwl" (UniqueName: "kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl") pod "network-check-target-mvhbl" (UID: "ed0d041c-4673-444b-8392-02de7cc0958f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:31.804108 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.804079 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndln\" (UniqueName: \"kubernetes.io/projected/30528697-a751-4fec-9d6b-d7446534904e-kube-api-access-wndln\") pod \"tuned-4c984\" (UID: \"30528697-a751-4fec-9d6b-d7446534904e\") " pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.806450 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.806418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l748\" (UniqueName: \"kubernetes.io/projected/944b5648-71ef-4980-927f-351334db589e-kube-api-access-7l748\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:31.806580 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.806556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzvf\" (UniqueName: \"kubernetes.io/projected/e1d102fd-e6fa-47af-8ec8-b6ce4a292be3-kube-api-access-2pzvf\") pod \"iptables-alerter-9gnd7\" (UID: \"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3\") " pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:31.808410 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.808384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prld\" (UniqueName: \"kubernetes.io/projected/8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd-kube-api-access-7prld\") pod \"multus-ppcww\" (UID: \"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd\") " pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.809076 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.809049 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqg4q\" (UniqueName: \"kubernetes.io/projected/ad37714e-ec26-404b-8b83-76e78c1d12a5-kube-api-access-tqg4q\") pod \"multus-additional-cni-plugins-jvhqn\" (UID: \"ad37714e-ec26-404b-8b83-76e78c1d12a5\") " pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.812624 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.812598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8q6z\" (UniqueName: \"kubernetes.io/projected/8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e-kube-api-access-x8q6z\") pod \"aws-ebs-csi-driver-node-t6hr8\" (UID: \"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:31.814666 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.814636 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkrr\" (UniqueName: \"kubernetes.io/projected/0f12a9ba-2874-4c63-803e-b40f109453ac-kube-api-access-qbkrr\") pod \"node-ca-kcxpt\" (UID: \"0f12a9ba-2874-4c63-803e-b40f109453ac\") " pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.876969 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.876933 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-ovn\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.876985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-kubelet\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-node-log\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877025 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/59c257de-f095-4753-adb3-14789e3b440d-ovn-node-metrics-cert\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877042 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-log-socket\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-env-overrides\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-run-netns\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877123 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-systemd\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877149 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-node-log\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877171 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-systemd\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877170 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-log-socket\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877184 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-ovn\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877216 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-kubelet\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-cni-netd\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-run-netns\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbnk\" (UniqueName: \"kubernetes.io/projected/59c257de-f095-4753-adb3-14789e3b440d-kube-api-access-7gbnk\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.877780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877748 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-cni-netd\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877842 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-env-overrides\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-ovnkube-config\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-run-ovn-kubernetes\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877919 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-run-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-run-ovn-kubernetes\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.877978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-var-lib-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878020 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-var-lib-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-systemd-units\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878050 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-systemd-units\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-ovnkube-script-lib\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-slash\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878190 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-slash\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-etc-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-cni-bin\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878305 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-etc-openvswitch\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59c257de-f095-4753-adb3-14789e3b440d-host-cni-bin\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.878619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878373 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-ovnkube-config\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.879087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.878624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/59c257de-f095-4753-adb3-14789e3b440d-ovnkube-script-lib\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.881851 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.881827 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/59c257de-f095-4753-adb3-14789e3b440d-ovn-node-metrics-cert\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.890347 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.890306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbnk\" (UniqueName: \"kubernetes.io/projected/59c257de-f095-4753-adb3-14789e3b440d-kube-api-access-7gbnk\") pod \"ovnkube-node-h2jpq\" (UID: \"59c257de-f095-4753-adb3-14789e3b440d\") " pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:31.951347 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.951057 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:31.959150 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.959097 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcxpt" Sep 10 16:17:31.965970 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.965927 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ppcww" Sep 10 16:17:31.966439 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:31.966404 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda809f2d7_03d7_487d_a48e_4628fc304d2a.slice/crio-eab115441c13d02b431c93c83e23d175b3bdeed6417ddb49826364e02e81ff3b WatchSource:0}: Error finding container eab115441c13d02b431c93c83e23d175b3bdeed6417ddb49826364e02e81ff3b: Status 404 returned error can't find the container with id eab115441c13d02b431c93c83e23d175b3bdeed6417ddb49826364e02e81ff3b Sep 10 16:17:31.972944 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:31.972907 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f12a9ba_2874_4c63_803e_b40f109453ac.slice/crio-f1bcb9209ad6a830be9df4697055ce383cfe23cc4f8eea1f58db9101642ea7d6 WatchSource:0}: Error finding container f1bcb9209ad6a830be9df4697055ce383cfe23cc4f8eea1f58db9101642ea7d6: Status 404 returned error can't find the container with id f1bcb9209ad6a830be9df4697055ce383cfe23cc4f8eea1f58db9101642ea7d6 Sep 10 16:17:31.976152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.976127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jv76" Sep 10 16:17:31.977910 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:31.977843 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c743517_72d7_4bb4_bda1_eb8c9fbd2bbd.slice/crio-b00bbc88f5df1fb4576805cb2ac7f05eddd24c0d646208fba678475cf901dd67 WatchSource:0}: Error finding container b00bbc88f5df1fb4576805cb2ac7f05eddd24c0d646208fba678475cf901dd67: Status 404 returned error can't find the container with id b00bbc88f5df1fb4576805cb2ac7f05eddd24c0d646208fba678475cf901dd67 Sep 10 16:17:31.985380 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.985342 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" Sep 10 16:17:31.986445 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:31.986419 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff7f8c0_5a53_45bc_8b61_8abebb4bf30c.slice/crio-1a50cb346d7c42c4bafb5c2e0419520e1e6a95abed66396bb749e345cf549ca0 WatchSource:0}: Error finding container 1a50cb346d7c42c4bafb5c2e0419520e1e6a95abed66396bb749e345cf549ca0: Status 404 returned error can't find the container with id 1a50cb346d7c42c4bafb5c2e0419520e1e6a95abed66396bb749e345cf549ca0 Sep 10 16:17:31.994907 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:31.994861 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4c984" Sep 10 16:17:31.996102 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:31.996047 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad37714e_ec26_404b_8b83_76e78c1d12a5.slice/crio-5107df411205464703044996647489e12b4bd28b27de07d5fc49594523f8ade5 WatchSource:0}: Error finding container 5107df411205464703044996647489e12b4bd28b27de07d5fc49594523f8ade5: Status 404 returned error can't find the container with id 5107df411205464703044996647489e12b4bd28b27de07d5fc49594523f8ade5 Sep 10 16:17:32.004588 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:32.004557 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30528697_a751_4fec_9d6b_d7446534904e.slice/crio-61bc4dbb423f59f87e9c13b2ebe216e9130d840d6c3dd1427d4b3e26362b9c20 WatchSource:0}: Error finding container 61bc4dbb423f59f87e9c13b2ebe216e9130d840d6c3dd1427d4b3e26362b9c20: Status 404 returned error can't find the container with id 61bc4dbb423f59f87e9c13b2ebe216e9130d840d6c3dd1427d4b3e26362b9c20 Sep 10 16:17:32.004694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.004661 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" Sep 10 16:17:32.014671 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:32.014639 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec0cb62_8d6a_4a5c_b213_e6e5d033dd9e.slice/crio-e3164d0d839757df5ed8cc416200ab6fa4ec1d3d09c8036ea5eb6aef1296d28e WatchSource:0}: Error finding container e3164d0d839757df5ed8cc416200ab6fa4ec1d3d09c8036ea5eb6aef1296d28e: Status 404 returned error can't find the container with id e3164d0d839757df5ed8cc416200ab6fa4ec1d3d09c8036ea5eb6aef1296d28e Sep 10 16:17:32.016854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.016730 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9gnd7" Sep 10 16:17:32.025534 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.025512 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:32.033405 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:32.033376 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d102fd_e6fa_47af_8ec8_b6ce4a292be3.slice/crio-a18a0d3a38a09ccb2d2785f885273c2fed20754ec8f73888eaa25420472901ae WatchSource:0}: Error finding container a18a0d3a38a09ccb2d2785f885273c2fed20754ec8f73888eaa25420472901ae: Status 404 returned error can't find the container with id a18a0d3a38a09ccb2d2785f885273c2fed20754ec8f73888eaa25420472901ae Sep 10 16:17:32.038198 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:17:32.038089 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c257de_f095_4753_adb3_14789e3b440d.slice/crio-aeb5a3448c41d7534ceb33a2a01cbf0b9e5087e01a5f130b663046e24871d888 WatchSource:0}: Error finding container aeb5a3448c41d7534ceb33a2a01cbf0b9e5087e01a5f130b663046e24871d888: Status 404 returned error can't find the container with id aeb5a3448c41d7534ceb33a2a01cbf0b9e5087e01a5f130b663046e24871d888 Sep 10 16:17:32.280896 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.280800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:32.280896 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.280866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:32.281112 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:32.280982 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:32.281112 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:32.281085 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret podName:3cb5f5c7-9cca-44c3-9013-6fcc3788f877 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:33.281044464 +0000 UTC m=+4.007363396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret") pod "global-pull-secret-syncer-l849b" (UID: "3cb5f5c7-9cca-44c3-9013-6fcc3788f877") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:32.281112 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:32.280987 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:32.281281 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:32.281139 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs podName:944b5648-71ef-4980-927f-351334db589e nodeName:}" failed. No retries permitted until 2026-09-10 16:17:33.281127209 +0000 UTC m=+4.007446136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs") pod "network-metrics-daemon-hn92b" (UID: "944b5648-71ef-4980-927f-351334db589e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:32.382206 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.382161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:32.382419 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:32.382384 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:17:32.382419 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:32.382415 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:17:32.382512 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:32.382431 2562 projected.go:196] Error preparing data for projected volume kube-api-access-98xwl for pod openshift-network-diagnostics/network-check-target-mvhbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:32.382554 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:32.382509 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl podName:ed0d041c-4673-444b-8392-02de7cc0958f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:33.38248655 +0000 UTC m=+4.108805484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-98xwl" (UniqueName: "kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl") pod "network-check-target-mvhbl" (UID: "ed0d041c-4673-444b-8392-02de7cc0958f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:32.704905 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.704805 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 16:12:31 +0000 UTC" deadline="2028-06-12 18:50:55.633637238 +0000 UTC" Sep 10 16:17:32.704905 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.704849 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15386h33m22.928793412s" Sep 10 16:17:32.769969 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.769899 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcxpt" event={"ID":"0f12a9ba-2874-4c63-803e-b40f109453ac","Type":"ContainerStarted","Data":"f1bcb9209ad6a830be9df4697055ce383cfe23cc4f8eea1f58db9101642ea7d6"} Sep 10 16:17:32.772620 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.772587 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4544q" event={"ID":"a809f2d7-03d7-487d-a48e-4628fc304d2a","Type":"ContainerStarted","Data":"eab115441c13d02b431c93c83e23d175b3bdeed6417ddb49826364e02e81ff3b"} Sep 10 16:17:32.774419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.774369 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" event={"ID":"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e","Type":"ContainerStarted","Data":"e3164d0d839757df5ed8cc416200ab6fa4ec1d3d09c8036ea5eb6aef1296d28e"} Sep 10 16:17:32.776213 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.776184 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" event={"ID":"ad37714e-ec26-404b-8b83-76e78c1d12a5","Type":"ContainerStarted","Data":"5107df411205464703044996647489e12b4bd28b27de07d5fc49594523f8ade5"} Sep 10 16:17:32.777798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.777773 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jv76" event={"ID":"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c","Type":"ContainerStarted","Data":"1a50cb346d7c42c4bafb5c2e0419520e1e6a95abed66396bb749e345cf549ca0"} Sep 10 16:17:32.779431 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.779407 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"aeb5a3448c41d7534ceb33a2a01cbf0b9e5087e01a5f130b663046e24871d888"} Sep 10 16:17:32.781821 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.781796 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9gnd7" event={"ID":"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3","Type":"ContainerStarted","Data":"a18a0d3a38a09ccb2d2785f885273c2fed20754ec8f73888eaa25420472901ae"} Sep 10 16:17:32.783721 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.783660 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4c984" event={"ID":"30528697-a751-4fec-9d6b-d7446534904e","Type":"ContainerStarted","Data":"61bc4dbb423f59f87e9c13b2ebe216e9130d840d6c3dd1427d4b3e26362b9c20"} Sep 10 16:17:32.787858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:32.787830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ppcww" event={"ID":"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd","Type":"ContainerStarted","Data":"b00bbc88f5df1fb4576805cb2ac7f05eddd24c0d646208fba678475cf901dd67"} Sep 10 16:17:33.290286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.290240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:33.290483 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.290356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:33.290558 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.290510 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:33.290610 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.290578 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret podName:3cb5f5c7-9cca-44c3-9013-6fcc3788f877 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:35.29055867 +0000 UTC m=+6.016877601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret") pod "global-pull-secret-syncer-l849b" (UID: "3cb5f5c7-9cca-44c3-9013-6fcc3788f877") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:33.290950 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.290928 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:33.291023 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.290980 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs podName:944b5648-71ef-4980-927f-351334db589e nodeName:}" failed. No retries permitted until 2026-09-10 16:17:35.290967873 +0000 UTC m=+6.017286782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs") pod "network-metrics-daemon-hn92b" (UID: "944b5648-71ef-4980-927f-351334db589e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:33.391151 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.391104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:33.391367 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.391342 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:17:33.391367 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.391368 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:17:33.391493 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.391382 2562 projected.go:196] Error preparing data for projected volume kube-api-access-98xwl for pod openshift-network-diagnostics/network-check-target-mvhbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:33.391493 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.391449 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl podName:ed0d041c-4673-444b-8392-02de7cc0958f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:35.391428089 +0000 UTC m=+6.117747018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-98xwl" (UniqueName: "kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl") pod "network-check-target-mvhbl" (UID: "ed0d041c-4673-444b-8392-02de7cc0958f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:33.706078 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.705796 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 16:12:31 +0000 UTC" deadline="2028-02-11 14:30:07.845480605 +0000 UTC" Sep 10 16:17:33.706439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.706081 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12454h12m34.139406773s" Sep 10 16:17:33.752208 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.752170 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:33.752346 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.752300 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:33.752455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.752432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:33.752583 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.752560 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:33.752665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.752648 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:33.752752 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:33.752734 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:33.795455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.795415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" event={"ID":"e031a7be8eb499dcf51c16d6ce862b92","Type":"ContainerStarted","Data":"1472f9419b9b6699d243d91f52a63649a779155a5c2187f772267462879ecb4a"} Sep 10 16:17:33.812420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:33.812357 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-24.ec2.internal" podStartSLOduration=2.812337059 podStartE2EDuration="2.812337059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:33.812113155 +0000 UTC m=+4.538432086" watchObservedRunningTime="2026-09-10 16:17:33.812337059 +0000 UTC m=+4.538655991" Sep 10 16:17:34.804344 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:34.804307 2562 generic.go:356] "Generic (PLEG): container finished" podID="a5f367cb344977acd80fcfe8020385ad" containerID="547937a0af76eadb847a49539faac79d7c194e96eabf5972db32252ded2ec99a" exitCode=0 Sep 10 16:17:34.805366 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:34.805331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" event={"ID":"a5f367cb344977acd80fcfe8020385ad","Type":"ContainerDied","Data":"547937a0af76eadb847a49539faac79d7c194e96eabf5972db32252ded2ec99a"} Sep 10 16:17:35.307684 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:35.307641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:35.307853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:35.307720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:35.307926 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.307882 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:35.307982 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.307945 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs podName:944b5648-71ef-4980-927f-351334db589e nodeName:}" failed. No retries permitted until 2026-09-10 16:17:39.307927451 +0000 UTC m=+10.034246361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs") pod "network-metrics-daemon-hn92b" (UID: "944b5648-71ef-4980-927f-351334db589e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:35.308384 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.308349 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:35.308480 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.308406 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret podName:3cb5f5c7-9cca-44c3-9013-6fcc3788f877 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:39.308390045 +0000 UTC m=+10.034708955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret") pod "global-pull-secret-syncer-l849b" (UID: "3cb5f5c7-9cca-44c3-9013-6fcc3788f877") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:35.408655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:35.408617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:35.408881 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.408839 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:17:35.408881 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.408865 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:17:35.408881 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.408879 2562 projected.go:196] Error preparing data for projected volume kube-api-access-98xwl for pod openshift-network-diagnostics/network-check-target-mvhbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:35.409090 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.408939 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl podName:ed0d041c-4673-444b-8392-02de7cc0958f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:39.408920667 +0000 UTC m=+10.135239589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-98xwl" (UniqueName: "kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl") pod "network-check-target-mvhbl" (UID: "ed0d041c-4673-444b-8392-02de7cc0958f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:35.756230 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:35.756153 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:35.756392 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.756310 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:35.756829 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:35.756807 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:35.756974 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.756951 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:35.757092 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:35.757078 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:35.757232 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:35.757212 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:37.751715 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:37.751678 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:37.752216 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:37.751678 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:37.752216 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:37.751807 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:37.752216 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:37.751813 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:37.752216 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:37.751916 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:37.752216 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:37.752001 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:39.346019 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:39.345706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:39.346019 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:39.345791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:39.346019 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.345853 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:39.346019 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.345927 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:39.346019 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.345931 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret podName:3cb5f5c7-9cca-44c3-9013-6fcc3788f877 nodeName:}" failed. No retries permitted until 2026-09-10 16:17:47.345909315 +0000 UTC m=+18.072228238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret") pod "global-pull-secret-syncer-l849b" (UID: "3cb5f5c7-9cca-44c3-9013-6fcc3788f877") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:39.346019 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.345994 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs podName:944b5648-71ef-4980-927f-351334db589e nodeName:}" failed. No retries permitted until 2026-09-10 16:17:47.345976721 +0000 UTC m=+18.072295628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs") pod "network-metrics-daemon-hn92b" (UID: "944b5648-71ef-4980-927f-351334db589e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:39.446391 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:39.446347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:39.446587 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.446554 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:17:39.446587 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.446579 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:17:39.446719 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.446593 2562 projected.go:196] Error preparing data for projected volume kube-api-access-98xwl for pod openshift-network-diagnostics/network-check-target-mvhbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:39.446719 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.446654 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl podName:ed0d041c-4673-444b-8392-02de7cc0958f nodeName:}" failed. No retries permitted until 2026-09-10 16:17:47.446635469 +0000 UTC m=+18.172954380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-98xwl" (UniqueName: "kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl") pod "network-check-target-mvhbl" (UID: "ed0d041c-4673-444b-8392-02de7cc0958f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:39.752751 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:39.752674 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:39.752910 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.752800 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:39.753338 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:39.753062 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:39.753338 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.753177 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:39.753338 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:39.753222 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:39.753338 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:39.753303 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:41.751740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:41.751702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:41.752346 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:41.751832 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:41.752346 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:41.752278 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:41.752441 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:41.752371 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:41.752642 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:41.752509 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:41.752642 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:41.752622 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:43.751740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:43.751702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:43.752228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:43.751702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:43.753050 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:43.751708 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:43.753050 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:43.752367 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:43.753050 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:43.752615 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:43.753050 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:43.752733 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:45.750869 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:45.750837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:45.750869 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:45.750864 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:45.751412 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:45.750838 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:45.751412 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:45.750948 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:45.751412 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:45.751054 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:45.751412 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:45.751131 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:47.409357 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:47.409309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:47.409857 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:47.409394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:47.409857 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.409490 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:47.409857 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.409529 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:47.409857 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.409572 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret podName:3cb5f5c7-9cca-44c3-9013-6fcc3788f877 nodeName:}" failed. No retries permitted until 2026-09-10 16:18:03.409551971 +0000 UTC m=+34.135870898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret") pod "global-pull-secret-syncer-l849b" (UID: "3cb5f5c7-9cca-44c3-9013-6fcc3788f877") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:17:47.409857 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.409595 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs podName:944b5648-71ef-4980-927f-351334db589e nodeName:}" failed. No retries permitted until 2026-09-10 16:18:03.409583639 +0000 UTC m=+34.135902553 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs") pod "network-metrics-daemon-hn92b" (UID: "944b5648-71ef-4980-927f-351334db589e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:17:47.510182 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:47.510148 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:47.510358 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.510305 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:17:47.510358 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.510326 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:17:47.510358 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.510339 2562 projected.go:196] Error preparing data for projected volume kube-api-access-98xwl for pod openshift-network-diagnostics/network-check-target-mvhbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:47.510487 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.510403 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl podName:ed0d041c-4673-444b-8392-02de7cc0958f nodeName:}" failed. No retries permitted until 2026-09-10 16:18:03.510388047 +0000 UTC m=+34.236706974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-98xwl" (UniqueName: "kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl") pod "network-check-target-mvhbl" (UID: "ed0d041c-4673-444b-8392-02de7cc0958f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:17:47.751803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:47.751763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:47.751982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:47.751763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:47.751982 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.751911 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:47.751982 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.751973 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:47.752151 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:47.751763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:47.752151 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:47.752107 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:49.751715 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:49.751675 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:49.752155 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:49.751778 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:49.752155 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:49.751777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:49.752155 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:49.751844 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:49.752155 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:49.751853 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:49.752155 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:49.751969 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:50.839167 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.838816 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcxpt" event={"ID":"0f12a9ba-2874-4c63-803e-b40f109453ac","Type":"ContainerStarted","Data":"b80b723f07cd3cdc2021af83ed8f621e984f975144d75c8fc2e4a5c29d519629"} Sep 10 16:17:50.840376 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.840353 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4544q" event={"ID":"a809f2d7-03d7-487d-a48e-4628fc304d2a","Type":"ContainerStarted","Data":"6f81190175e94889e359c9d0f956eae831666f773eb028c2e6a5131cc49eb174"} Sep 10 16:17:50.842436 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.842415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" event={"ID":"a5f367cb344977acd80fcfe8020385ad","Type":"ContainerStarted","Data":"6d4e1bb92bae0408cfd50e3da51a53089cdf260db892cbf6272047bdca7c6e45"} Sep 10 16:17:50.843937 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.843918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" event={"ID":"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e","Type":"ContainerStarted","Data":"25585398a793acdc3549b89f201f3b474d97fef910523ea2070ed3066fdb7b85"} Sep 10 16:17:50.845561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.845533 2562 generic.go:356] "Generic (PLEG): container finished" podID="ad37714e-ec26-404b-8b83-76e78c1d12a5" containerID="6faddbd398d2712049db6b4dee0206b7c23ba53246bea3c5b642c6c420e6719e" exitCode=0 Sep 10 16:17:50.845629 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.845611 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" event={"ID":"ad37714e-ec26-404b-8b83-76e78c1d12a5","Type":"ContainerDied","Data":"6faddbd398d2712049db6b4dee0206b7c23ba53246bea3c5b642c6c420e6719e"} Sep 10 16:17:50.847562 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.847519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jv76" event={"ID":"dff7f8c0-5a53-45bc-8b61-8abebb4bf30c","Type":"ContainerStarted","Data":"ae5caf869cd3ad45b31b25da8a55ca651ec2a09486e9f89213dcbd7fb90f139c"} Sep 10 16:17:50.849910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.849896 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:17:50.850307 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.850282 2562 generic.go:356] "Generic (PLEG): container finished" podID="59c257de-f095-4753-adb3-14789e3b440d" containerID="7cff902d06b8924041e4f27879c0b59de0cebe6b1bc0ae03b0ff72925fbe5a0d" exitCode=1 Sep 10 16:17:50.850382 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.850356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"a45397aba888f41a8f7306cd7578c87a06a199a722db188cde4b1f2e2449b30c"} Sep 10 16:17:50.850461 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.850445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerDied","Data":"7cff902d06b8924041e4f27879c0b59de0cebe6b1bc0ae03b0ff72925fbe5a0d"} Sep 10 16:17:50.850521 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.850505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"8da8e5d4a4bd46accb952645333d596ef46fbf60bcd99fda097249cf60009d81"} Sep 10 16:17:50.852022 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.851998 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4c984" event={"ID":"30528697-a751-4fec-9d6b-d7446534904e","Type":"ContainerStarted","Data":"63da4589e77fa5db41cc7aa0c6c0f66a6fa2c11cfd16b554309004477c66a3ca"} Sep 10 16:17:50.853712 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.853685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ppcww" event={"ID":"8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd","Type":"ContainerStarted","Data":"5edff803cb936ee823e8b5b7318cf00f51a1b7133235c47b8ee9087690428e50"} Sep 10 16:17:50.856854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.856814 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kcxpt" podStartSLOduration=2.4879721249999998 podStartE2EDuration="20.856802072s" totalImagesPullingTime="18.368829947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:50.856236427 +0000 UTC m=+21.582555357" watchObservedRunningTime="2026-09-10 16:17:50.856802072 +0000 UTC m=+21.583121001" Sep 10 16:17:50.873446 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.873397 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-24.ec2.internal" podStartSLOduration=19.873380792 podStartE2EDuration="19.873380792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:50.873114599 +0000 UTC m=+21.599433530" watchObservedRunningTime="2026-09-10 16:17:50.873380792 +0000 UTC m=+21.599699721" Sep 10 16:17:50.892805 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.892755 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4544q" podStartSLOduration=2.502444343 podStartE2EDuration="20.892736712s" totalImagesPullingTime="18.390292369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:50.89273829 +0000 UTC m=+21.619057219" watchObservedRunningTime="2026-09-10 16:17:50.892736712 +0000 UTC m=+21.619055653" Sep 10 16:17:50.910354 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.910299 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8jv76" podStartSLOduration=2.5359534 podStartE2EDuration="20.910283573s" totalImagesPullingTime="18.374330173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:50.909968343 +0000 UTC m=+21.636287273" watchObservedRunningTime="2026-09-10 16:17:50.910283573 +0000 UTC m=+21.636602502" Sep 10 16:17:50.953410 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.953353 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4c984" podStartSLOduration=2.599269775 podStartE2EDuration="20.953335424s" totalImagesPullingTime="18.354065649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:50.953236196 +0000 UTC m=+21.679555153" watchObservedRunningTime="2026-09-10 16:17:50.953335424 +0000 UTC m=+21.679654357" Sep 10 16:17:50.958228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.958205 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:50.974008 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:50.973961 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ppcww" podStartSLOduration=2.274904352 podStartE2EDuration="20.973944853s" totalImagesPullingTime="18.699040501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:50.973341387 +0000 UTC m=+21.699660320" watchObservedRunningTime="2026-09-10 16:17:50.973944853 +0000 UTC m=+21.700263783" Sep 10 16:17:51.101078 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.101021 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:51.200284 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.200236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:51.751583 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.751555 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:51.751739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.751555 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:51.751739 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:51.751689 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:51.751861 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:51.751755 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:51.751861 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.751793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:51.751954 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:51.751885 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:51.859554 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.859524 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:17:51.860236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.859978 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"cb14f1acb682ed7760690104574a4c5644dd38bf7346e0e54ff7af192aaedb8f"} Sep 10 16:17:51.860236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.860086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"cb5fdfcbc8320e9cb330255fdd996221ed726cdc928d37ce0ad19e5254c36419"} Sep 10 16:17:51.860236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:51.860137 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"608802a625156535e60c345b1f120a6e0d578475ced65a6d616fe8ea987f336e"} Sep 10 16:17:52.002878 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.002798 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 16:17:52.372091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.371996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:52.449498 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.449471 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:52.854756 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.854646 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T16:17:52.002820973Z","UUID":"bad12906-30d2-467a-8aa0-8fdb40cbb5c0","Handler":null,"Name":"","Endpoint":""} Sep 10 16:17:52.857817 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.857790 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 16:17:52.857953 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.857828 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 16:17:52.864401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.864370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" event={"ID":"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e","Type":"ContainerStarted","Data":"b6e067f5f5c4d3f8745496acc2d071468c2df1007fb4e9d7c3e68b42e5a038f9"} Sep 10 16:17:52.866357 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.866328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9gnd7" event={"ID":"e1d102fd-e6fa-47af-8ec8-b6ce4a292be3","Type":"ContainerStarted","Data":"9fd7f8866ca75606ca640710d773a83abcd3cb38c4dc780c4ba0b7ec3b020c57"} Sep 10 16:17:52.871131 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.871103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:52.884657 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:52.884608 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9gnd7" podStartSLOduration=4.576494569 podStartE2EDuration="22.884592775s" totalImagesPullingTime="18.308098206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:52.884496866 +0000 UTC m=+23.610815820" watchObservedRunningTime="2026-09-10 16:17:52.884592775 +0000 UTC m=+23.610911707" Sep 10 16:17:53.072988 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.072949 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:53.172991 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.172761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:53.346552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.346523 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jv76_dff7f8c0-5a53-45bc-8b61-8abebb4bf30c/dns-node-resolver/0.log" Sep 10 16:17:53.353545 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.353504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:53.422020 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.421985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:53.483434 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.483403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:53.668162 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.668082 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:53.668745 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.668713 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:53.750973 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.750939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:53.751165 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.750938 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:53.751165 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:53.751088 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:53.751165 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:53.751122 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:53.751165 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.750939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:53.751383 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:53.751212 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:53.871326 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.871292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:53.872149 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.872072 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" event={"ID":"8ec0cb62-8d6a-4a5c-b213-e6e5d033dd9e","Type":"ContainerStarted","Data":"aacb11c5d0887807b4088f1b3ec61428ec48d1f3d81da736d3ec458cdb161962"} Sep 10 16:17:53.876935 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.876915 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:17:53.877505 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.877479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"8051e5090a22a95c878636b9d63b1cc5d4e0d204544067ca91f6a6e695afd665"} Sep 10 16:17:53.877893 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.877875 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:53.878377 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.878362 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4544q" Sep 10 16:17:53.912047 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.911978 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6hr8" podStartSLOduration=2.934957856 podStartE2EDuration="23.911961025s" totalImagesPullingTime="20.977003169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:53.894734429 +0000 UTC m=+24.621053356" watchObservedRunningTime="2026-09-10 16:17:53.911961025 +0000 UTC m=+24.638279954" Sep 10 16:17:53.972523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:53.972441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:54.075165 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:54.075131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:54.177459 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:54.177418 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:54.238782 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:54.238700 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:54.730760 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:54.730727 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcxpt_0f12a9ba-2874-4c63-803e-b40f109453ac/node-ca/0.log" Sep 10 16:17:55.751701 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:55.751664 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:55.751701 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:55.751681 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:55.752252 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:55.751664 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:55.752252 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:55.751790 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:55.752252 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:55.751858 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:55.752252 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:55.751984 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:55.791622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:55.791510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:55.865762 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:55.865724 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:55.885407 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:55.885386 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:17:55.885842 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:55.885816 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"5704c7c2cb1aac6f68f961d1a97e5eec4fc4c82e5edddddbfe561d9c70099e7d"} Sep 10 16:17:55.886295 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:55.886265 2562 scope.go:117] "RemoveContainer" containerID="7cff902d06b8924041e4f27879c0b59de0cebe6b1bc0ae03b0ff72925fbe5a0d" Sep 10 16:17:56.782447 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.782266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:56.889743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.889710 2562 generic.go:356] "Generic (PLEG): container finished" podID="ad37714e-ec26-404b-8b83-76e78c1d12a5" containerID="392409298cf944cb93c9d81d782cddef1b18f2f4f0f51b0e8cf6a67e90db09ef" exitCode=0 Sep 10 16:17:56.889933 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.889802 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" event={"ID":"ad37714e-ec26-404b-8b83-76e78c1d12a5","Type":"ContainerDied","Data":"392409298cf944cb93c9d81d782cddef1b18f2f4f0f51b0e8cf6a67e90db09ef"} Sep 10 16:17:56.893934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.893914 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:17:56.894297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.894270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" event={"ID":"59c257de-f095-4753-adb3-14789e3b440d","Type":"ContainerStarted","Data":"74239ab07d94d89e343c5793c2d1016f50a0a514af28c05a47c7e382dda582f1"} Sep 10 16:17:56.894513 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.894487 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:56.894618 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.894568 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:56.894686 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.894622 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:56.910230 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.910160 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:56.910351 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.910260 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:17:56.950621 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:56.950553 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" podStartSLOduration=8.604681372 podStartE2EDuration="26.950531417s" totalImagesPullingTime="18.345850045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:56.949718807 +0000 UTC m=+27.676037735" watchObservedRunningTime="2026-09-10 16:17:56.950531417 +0000 UTC m=+27.676850350" Sep 10 16:17:57.644821 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.644788 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l849b"] Sep 10 16:17:57.644974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.644958 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:57.645116 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:57.645091 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:57.648251 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.648223 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hn92b"] Sep 10 16:17:57.648350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.648300 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mvhbl"] Sep 10 16:17:57.648409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.648352 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:57.648409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.648366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:57.648478 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:57.648449 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:57.648539 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:57.648522 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:57.873059 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.873010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:57.899129 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.899100 2562 generic.go:356] "Generic (PLEG): container finished" podID="ad37714e-ec26-404b-8b83-76e78c1d12a5" containerID="3d212b3eeaa90d1d3572ccdbd585b08305b62ac3dfb962461ae49c00c749ec87" exitCode=0 Sep 10 16:17:57.899379 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:57.899195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" event={"ID":"ad37714e-ec26-404b-8b83-76e78c1d12a5","Type":"ContainerDied","Data":"3d212b3eeaa90d1d3572ccdbd585b08305b62ac3dfb962461ae49c00c749ec87"} Sep 10 16:17:58.231832 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:58.231799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:58.337281 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:58.337251 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:58.433152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:58.433108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:58.534187 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:58.534160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:58.644133 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:58.644104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:58.903439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:58.903349 2562 generic.go:356] "Generic (PLEG): container finished" podID="ad37714e-ec26-404b-8b83-76e78c1d12a5" containerID="a4104fd70c8e051dfa886dd90672ffe3ca8fb993fba215967f6d72d78e436505" exitCode=0 Sep 10 16:17:58.903838 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:58.903434 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" event={"ID":"ad37714e-ec26-404b-8b83-76e78c1d12a5","Type":"ContainerDied","Data":"a4104fd70c8e051dfa886dd90672ffe3ca8fb993fba215967f6d72d78e436505"} Sep 10 16:17:58.933421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:58.933393 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:59.669670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:59.669587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:59.752393 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:59.752363 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:17:59.752575 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:59.752467 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:17:59.752575 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:59.752546 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:17:59.752695 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:59.752607 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:17:59.752695 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:59.752629 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:17:59.752790 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:17:59.752712 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:17:59.770986 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:59.770947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:59.927302 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:17:59.927209 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:00.090686 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:00.090644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:00.188334 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:00.188245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:00.351133 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:00.351094 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:00.755340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:00.755304 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:01.140069 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:01.139977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:01.742619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:01.742529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:01.751671 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:01.751637 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:01.751831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:01.751684 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:01.751831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:01.751733 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:01.751831 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:01.751748 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:01.751999 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:01.751834 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:01.751999 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:01.751923 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:01.845953 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:01.845906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:01.943353 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:01.943319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:02.068145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:02.068097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:02.368710 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:02.368631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:02.534763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:02.534732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:02.956638 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:02.956589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:03.031967 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.031932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:03.173299 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.173269 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:03.429586 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.429553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:03.430071 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.429608 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:03.430071 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.429701 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 16:18:03.430071 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.429723 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:18:03.430071 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.429751 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret podName:3cb5f5c7-9cca-44c3-9013-6fcc3788f877 nodeName:}" failed. No retries permitted until 2026-09-10 16:18:35.429738199 +0000 UTC m=+66.156057120 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret") pod "global-pull-secret-syncer-l849b" (UID: "3cb5f5c7-9cca-44c3-9013-6fcc3788f877") : object "kube-system"/"original-pull-secret" not registered Sep 10 16:18:03.430071 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.429793 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs podName:944b5648-71ef-4980-927f-351334db589e nodeName:}" failed. No retries permitted until 2026-09-10 16:18:35.429775459 +0000 UTC m=+66.156094381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs") pod "network-metrics-daemon-hn92b" (UID: "944b5648-71ef-4980-927f-351334db589e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 16:18:03.529958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.529918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:03.530135 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.530069 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 16:18:03.530135 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.530087 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 16:18:03.530135 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.530098 2562 projected.go:196] Error preparing data for projected volume kube-api-access-98xwl for pod openshift-network-diagnostics/network-check-target-mvhbl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:18:03.530255 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.530155 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl podName:ed0d041c-4673-444b-8392-02de7cc0958f nodeName:}" failed. No retries permitted until 2026-09-10 16:18:35.530140694 +0000 UTC m=+66.256459602 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-98xwl" (UniqueName: "kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl") pod "network-check-target-mvhbl" (UID: "ed0d041c-4673-444b-8392-02de7cc0958f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 16:18:03.741021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.740991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:03.751222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.751191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:03.751359 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.751191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:03.751359 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.751335 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:03.751463 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.751191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:03.751463 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.751400 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:03.751535 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:03.751514 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:03.918897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.918803 2562 generic.go:356] "Generic (PLEG): container finished" podID="ad37714e-ec26-404b-8b83-76e78c1d12a5" containerID="ce50d6a7aac9500993178e99f6985a7126867e5ed029543d0194528c6c4b6b66" exitCode=0 Sep 10 16:18:03.918897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.918853 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" event={"ID":"ad37714e-ec26-404b-8b83-76e78c1d12a5","Type":"ContainerDied","Data":"ce50d6a7aac9500993178e99f6985a7126867e5ed029543d0194528c6c4b6b66"} Sep 10 16:18:03.932224 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:03.932197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:04.021238 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:04.021214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:04.139944 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:04.139918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:04.338342 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:04.338310 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:04.464332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:04.464299 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:04.563713 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:04.563680 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:04.924433 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:04.924405 2562 generic.go:356] "Generic (PLEG): container finished" podID="ad37714e-ec26-404b-8b83-76e78c1d12a5" containerID="4c3c28c558815df93e64868defc842386f5b56c67f43f8f9b7d7887b1c1154f9" exitCode=0 Sep 10 16:18:04.924594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:04.924443 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" event={"ID":"ad37714e-ec26-404b-8b83-76e78c1d12a5","Type":"ContainerDied","Data":"4c3c28c558815df93e64868defc842386f5b56c67f43f8f9b7d7887b1c1154f9"} Sep 10 16:18:05.734105 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:05.733889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:05.751092 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:05.751066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:05.751200 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:05.751066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:05.751235 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:05.751207 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:05.751270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:05.751242 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:05.751337 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:05.751319 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:05.751406 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:05.751391 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:05.929860 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:05.929827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" event={"ID":"ad37714e-ec26-404b-8b83-76e78c1d12a5","Type":"ContainerStarted","Data":"e6855af45ca7e0f0036929b14130b94200611cbb23de9edbea37ddeb85ade61a"} Sep 10 16:18:05.937209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:05.937176 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:05.955317 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:05.955266 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jvhqn" podStartSLOduration=6.558876452 podStartE2EDuration="35.955252123s" totalImagesPullingTime="29.396375671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:05.953905286 +0000 UTC m=+36.680224216" watchObservedRunningTime="2026-09-10 16:18:05.955252123 +0000 UTC m=+36.681571053" Sep 10 16:18:06.133444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:06.133402 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:06.340194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:06.340164 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:06.537212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:06.537177 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:07.061086 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:07.061053 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:07.536790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:07.536758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:07.735382 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:07.735294 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:07.751406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:07.751370 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:07.751406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:07.751396 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:07.751569 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:07.751370 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:07.751605 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:07.751570 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:07.751707 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:07.751689 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:07.751772 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:07.751760 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:07.931714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:07.931681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:08.338377 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:08.338343 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:08.477929 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:08.477895 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:08.713115 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:08.713013 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:08.814544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:08.814512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:08.937089 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:08.937057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:09.752001 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:09.751962 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:09.752478 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:09.752067 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:09.752478 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:09.752089 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:09.752478 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:09.752069 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:09.752478 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:09.752152 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:09.752478 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:09.752236 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:11.751241 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:11.751195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:11.751679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:11.751195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:11.751679 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:11.751320 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:11.751679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:11.751341 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:11.751679 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:11.751392 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:11.751679 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:11.751456 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:13.750786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:13.750747 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:13.750786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:13.750765 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:13.751338 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:13.750765 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:13.751338 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:13.750869 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:13.751338 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:13.750943 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:13.751338 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:13.751003 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:14.672166 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:14.672126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:14.743064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:14.743020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:15.751019 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:15.750994 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:15.751410 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:15.751110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:15.751410 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:15.751118 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:15.751410 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:15.751138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:15.751410 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:15.751200 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:15.751410 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:15.751269 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:16.148874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:16.148767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:16.256287 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:16.256248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:16.352082 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:16.352018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:16.452507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:16.452406 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:16.615460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:16.615426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:16.809170 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:16.809139 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:16.878318 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:16.878282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:17.217239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.217164 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:17.320854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.320818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:17.422751 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.422703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:17.482940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.482863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:17.750971 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.750945 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:17.751181 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.750945 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:17.751181 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:17.751102 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvhbl" podUID="ed0d041c-4673-444b-8392-02de7cc0958f" Sep 10 16:18:17.751181 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.750945 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:17.751181 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:17.751144 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn92b" podUID="944b5648-71ef-4980-927f-351334db589e" Sep 10 16:18:17.751339 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:17.751196 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l849b" podUID="3cb5f5c7-9cca-44c3-9013-6fcc3788f877" Sep 10 16:18:17.873435 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.873377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:17.974479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:17.974441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:18.075574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:18.075489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:18.164394 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:18.164364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:18.454739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:18.454669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:18.904692 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:18.904653 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:19.119907 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.119883 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:19.279243 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.279215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:19.436053 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.436004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:19.535336 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.535229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:19.600009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.599983 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-24.ec2.internal" event="NodeReady" Sep 10 16:18:19.600181 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.600123 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 16:18:19.662280 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.662229 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tmhw5"] Sep 10 16:18:19.683184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.683151 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tmhw5" Sep 10 16:18:19.683505 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.683150 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jmfkp"] Sep 10 16:18:19.686855 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.686831 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 16:18:19.686855 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.686853 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kr9bx\"" Sep 10 16:18:19.687073 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.686887 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 16:18:19.687073 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.686940 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 16:18:19.707133 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.707102 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tmhw5"] Sep 10 16:18:19.707133 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.707138 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jmfkp"] Sep 10 16:18:19.707311 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.707161 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sb6w5"] Sep 10 16:18:19.707311 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.707232 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.710828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.710799 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 16:18:19.711383 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.711217 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 16:18:19.712030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.712004 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 16:18:19.712468 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.712193 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 16:18:19.712724 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.712701 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9wnm\"" Sep 10 16:18:19.723103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.723081 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sb6w5"] Sep 10 16:18:19.723288 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.723205 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.725876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.725851 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 16:18:19.725876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.725856 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wwcbb\"" Sep 10 16:18:19.726292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.726275 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 16:18:19.735643 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.735598 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:19.754029 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.754002 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:19.754176 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.754100 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:19.754276 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.754254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:19.757515 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.757496 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 16:18:19.757760 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.757746 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 16:18:19.757898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.757884 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j8r7d\"" Sep 10 16:18:19.757988 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.757975 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 16:18:19.758067 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.758028 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 16:18:19.758351 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.758338 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htvxx\"" Sep 10 16:18:19.838439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.838356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:19.841378 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8479af56-72f9-4e42-88ff-2f87408a9455-data-volume\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.841440 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzzbw\" (UniqueName: \"kubernetes.io/projected/f3531b68-1218-45f2-8e39-c7b7b948ecc7-kube-api-access-qzzbw\") pod \"ingress-canary-tmhw5\" (UID: \"f3531b68-1218-45f2-8e39-c7b7b948ecc7\") " pod="openshift-ingress-canary/ingress-canary-tmhw5" Sep 10 16:18:19.841440 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2311967-fbe1-4acc-8e9b-1145db261669-config-volume\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.841440 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841422 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2311967-fbe1-4acc-8e9b-1145db261669-metrics-tls\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.841557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3531b68-1218-45f2-8e39-c7b7b948ecc7-cert\") pod \"ingress-canary-tmhw5\" (UID: \"f3531b68-1218-45f2-8e39-c7b7b948ecc7\") " pod="openshift-ingress-canary/ingress-canary-tmhw5" Sep 10 16:18:19.841557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841494 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2311967-fbe1-4acc-8e9b-1145db261669-tmp-dir\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.841557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8479af56-72f9-4e42-88ff-2f87408a9455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.841557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8479af56-72f9-4e42-88ff-2f87408a9455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.841673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cjm\" (UniqueName: \"kubernetes.io/projected/e2311967-fbe1-4acc-8e9b-1145db261669-kube-api-access-n4cjm\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.841673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpgq\" (UniqueName: \"kubernetes.io/projected/8479af56-72f9-4e42-88ff-2f87408a9455-kube-api-access-kbpgq\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.841673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.841611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8479af56-72f9-4e42-88ff-2f87408a9455-crio-socket\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.935224 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.935192 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:19.942271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8479af56-72f9-4e42-88ff-2f87408a9455-data-volume\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.942340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzzbw\" (UniqueName: \"kubernetes.io/projected/f3531b68-1218-45f2-8e39-c7b7b948ecc7-kube-api-access-qzzbw\") pod \"ingress-canary-tmhw5\" (UID: \"f3531b68-1218-45f2-8e39-c7b7b948ecc7\") " pod="openshift-ingress-canary/ingress-canary-tmhw5" Sep 10 16:18:19.942340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2311967-fbe1-4acc-8e9b-1145db261669-config-volume\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.942419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2311967-fbe1-4acc-8e9b-1145db261669-metrics-tls\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.942558 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942540 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3531b68-1218-45f2-8e39-c7b7b948ecc7-cert\") pod \"ingress-canary-tmhw5\" (UID: \"f3531b68-1218-45f2-8e39-c7b7b948ecc7\") " pod="openshift-ingress-canary/ingress-canary-tmhw5" Sep 10 16:18:19.942641 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2311967-fbe1-4acc-8e9b-1145db261669-tmp-dir\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.942641 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8479af56-72f9-4e42-88ff-2f87408a9455-data-volume\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.942641 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8479af56-72f9-4e42-88ff-2f87408a9455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.942833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942638 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8479af56-72f9-4e42-88ff-2f87408a9455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.942886 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cjm\" (UniqueName: \"kubernetes.io/projected/e2311967-fbe1-4acc-8e9b-1145db261669-kube-api-access-n4cjm\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.942886 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpgq\" (UniqueName: \"kubernetes.io/projected/8479af56-72f9-4e42-88ff-2f87408a9455-kube-api-access-kbpgq\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.942989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.942899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8479af56-72f9-4e42-88ff-2f87408a9455-crio-socket\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.943058 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.943016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2311967-fbe1-4acc-8e9b-1145db261669-config-volume\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.943210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.943191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8479af56-72f9-4e42-88ff-2f87408a9455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.943260 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.943211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8479af56-72f9-4e42-88ff-2f87408a9455-crio-socket\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.943366 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.943348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2311967-fbe1-4acc-8e9b-1145db261669-tmp-dir\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.946274 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.946247 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8479af56-72f9-4e42-88ff-2f87408a9455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.946373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.946283 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2311967-fbe1-4acc-8e9b-1145db261669-metrics-tls\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.946627 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.946608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3531b68-1218-45f2-8e39-c7b7b948ecc7-cert\") pod \"ingress-canary-tmhw5\" (UID: \"f3531b68-1218-45f2-8e39-c7b7b948ecc7\") " pod="openshift-ingress-canary/ingress-canary-tmhw5" Sep 10 16:18:19.950765 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.950738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzzbw\" (UniqueName: \"kubernetes.io/projected/f3531b68-1218-45f2-8e39-c7b7b948ecc7-kube-api-access-qzzbw\") pod \"ingress-canary-tmhw5\" (UID: \"f3531b68-1218-45f2-8e39-c7b7b948ecc7\") " pod="openshift-ingress-canary/ingress-canary-tmhw5" Sep 10 16:18:19.951299 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.951272 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cjm\" (UniqueName: \"kubernetes.io/projected/e2311967-fbe1-4acc-8e9b-1145db261669-kube-api-access-n4cjm\") pod \"dns-default-sb6w5\" (UID: \"e2311967-fbe1-4acc-8e9b-1145db261669\") " pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:19.951636 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.951605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpgq\" (UniqueName: \"kubernetes.io/projected/8479af56-72f9-4e42-88ff-2f87408a9455-kube-api-access-kbpgq\") pod \"insights-runtime-extractor-jmfkp\" (UID: \"8479af56-72f9-4e42-88ff-2f87408a9455\") " pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:19.992192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:19.992139 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tmhw5" Sep 10 16:18:20.018783 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.018750 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jmfkp" Sep 10 16:18:20.031708 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.031684 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:20.298894 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.298860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tmhw5"] Sep 10 16:18:20.302331 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:20.302297 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3531b68_1218_45f2_8e39_c7b7b948ecc7.slice/crio-1054d97ba1bd5dc65a10a8ed54a446fe7a36c302f826f89bed9bc570435c125a WatchSource:0}: Error finding container 1054d97ba1bd5dc65a10a8ed54a446fe7a36c302f826f89bed9bc570435c125a: Status 404 returned error can't find the container with id 1054d97ba1bd5dc65a10a8ed54a446fe7a36c302f826f89bed9bc570435c125a Sep 10 16:18:20.305401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.305376 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jmfkp"] Sep 10 16:18:20.308546 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:20.308521 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8479af56_72f9_4e42_88ff_2f87408a9455.slice/crio-96428dd35e13ac75ac12faa9492880e9dbc655bfd2c3c669eaafeecdd26a0efb WatchSource:0}: Error finding container 96428dd35e13ac75ac12faa9492880e9dbc655bfd2c3c669eaafeecdd26a0efb: Status 404 returned error can't find the container with id 96428dd35e13ac75ac12faa9492880e9dbc655bfd2c3c669eaafeecdd26a0efb Sep 10 16:18:20.362214 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.362193 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sb6w5"] Sep 10 16:18:20.364111 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:20.364086 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2311967_fbe1_4acc_8e9b_1145db261669.slice/crio-f5d41725c0b6783985115edc822a57fa8314bd0e6e7fdf9a3dbde6e9bd437e25 WatchSource:0}: Error finding container f5d41725c0b6783985115edc822a57fa8314bd0e6e7fdf9a3dbde6e9bd437e25: Status 404 returned error can't find the container with id f5d41725c0b6783985115edc822a57fa8314bd0e6e7fdf9a3dbde6e9bd437e25 Sep 10 16:18:20.968072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.967921 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmfkp" event={"ID":"8479af56-72f9-4e42-88ff-2f87408a9455","Type":"ContainerStarted","Data":"a3aad86cccd3fab370a6cb9fad61fd6ebb3d99587060f6097cf251c3b4510087"} Sep 10 16:18:20.968072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.968000 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmfkp" event={"ID":"8479af56-72f9-4e42-88ff-2f87408a9455","Type":"ContainerStarted","Data":"96428dd35e13ac75ac12faa9492880e9dbc655bfd2c3c669eaafeecdd26a0efb"} Sep 10 16:18:20.969878 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.969837 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sb6w5" event={"ID":"e2311967-fbe1-4acc-8e9b-1145db261669","Type":"ContainerStarted","Data":"f5d41725c0b6783985115edc822a57fa8314bd0e6e7fdf9a3dbde6e9bd437e25"} Sep 10 16:18:20.972493 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:20.971561 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tmhw5" event={"ID":"f3531b68-1218-45f2-8e39-c7b7b948ecc7","Type":"ContainerStarted","Data":"1054d97ba1bd5dc65a10a8ed54a446fe7a36c302f826f89bed9bc570435c125a"} Sep 10 16:18:21.976654 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:21.976598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmfkp" event={"ID":"8479af56-72f9-4e42-88ff-2f87408a9455","Type":"ContainerStarted","Data":"6b11f3e9a78eca0db0a8569e8af4476faf565562846ad5c367d797bff58ae4cf"} Sep 10 16:18:23.681525 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:23.681473 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:23.785286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:23.785239 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:23.985337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:23.985250 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sb6w5" event={"ID":"e2311967-fbe1-4acc-8e9b-1145db261669","Type":"ContainerStarted","Data":"2419b902b2ded40857de466c5b0cd2dc7a7f901c84595ca22da5a3291cf69c40"} Sep 10 16:18:23.985337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:23.985302 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sb6w5" event={"ID":"e2311967-fbe1-4acc-8e9b-1145db261669","Type":"ContainerStarted","Data":"230965e01d443a21e7a85c2aff5644b76c1b3765e21c8673841e213210783dd9"} Sep 10 16:18:23.985561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:23.985370 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:23.986874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:23.986847 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tmhw5" event={"ID":"f3531b68-1218-45f2-8e39-c7b7b948ecc7","Type":"ContainerStarted","Data":"dd2ac0169c931ec310c26346e9fd84838136bb2b29027c58e39a84197b116fc9"} Sep 10 16:18:23.988911 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:23.988887 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmfkp" event={"ID":"8479af56-72f9-4e42-88ff-2f87408a9455","Type":"ContainerStarted","Data":"5832fa4f2195cbe6463ae926d809f0a5242fd9be3ebe779cfc6b94ccdaabad91"} Sep 10 16:18:24.006540 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:24.006490 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sb6w5" podStartSLOduration=2.143531585 podStartE2EDuration="5.006475987s" totalImagesPullingTime="2.862944402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:24.005582262 +0000 UTC m=+54.731901193" watchObservedRunningTime="2026-09-10 16:18:24.006475987 +0000 UTC m=+54.732794916" Sep 10 16:18:24.028016 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:24.027963 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jmfkp" podStartSLOduration=2.263357191 podStartE2EDuration="5.027944769s" totalImagesPullingTime="2.764587578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:18:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:24.027076981 +0000 UTC m=+54.753395911" watchObservedRunningTime="2026-09-10 16:18:24.027944769 +0000 UTC m=+54.754263737" Sep 10 16:18:24.044188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:24.044141 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tmhw5" podStartSLOduration=2.098960656 podStartE2EDuration="5.044124473s" totalImagesPullingTime="2.945163817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:24.043486516 +0000 UTC m=+54.769805456" watchObservedRunningTime="2026-09-10 16:18:24.044124473 +0000 UTC m=+54.770443403" Sep 10 16:18:27.254360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.254313 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-665655dc88-52ktb"] Sep 10 16:18:27.279830 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.279801 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-665655dc88-52ktb"] Sep 10 16:18:27.279967 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.279922 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.283076 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.283054 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 16:18:27.284490 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.284409 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 16:18:27.284490 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.284414 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qfpmt\"" Sep 10 16:18:27.284490 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.284414 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 16:18:27.284490 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.284477 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 16:18:27.284775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.284491 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 16:18:27.284775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.284522 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 16:18:27.284913 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.284897 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 16:18:27.298627 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.298593 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-serving-cert\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.298790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.298666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-service-ca\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.298790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.298709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6zj\" (UniqueName: \"kubernetes.io/projected/724b46cc-9941-422b-b2c8-05ed9414a031-kube-api-access-vd6zj\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.298790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.298759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-oauth-config\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.298790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.298780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-oauth-serving-cert\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.298991 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.298804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-console-config\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.399238 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.399196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-service-ca\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.399238 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.399243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6zj\" (UniqueName: \"kubernetes.io/projected/724b46cc-9941-422b-b2c8-05ed9414a031-kube-api-access-vd6zj\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.399472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.399266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-oauth-config\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.399472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.399281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-oauth-serving-cert\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.399472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.399298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-console-config\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.399472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.399336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-serving-cert\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.400018 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.399988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-oauth-serving-cert\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.400158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.399999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-console-config\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.400158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.400117 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-service-ca\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.401773 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.401749 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-oauth-config\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.401864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.401817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-serving-cert\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.408018 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.407990 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6zj\" (UniqueName: \"kubernetes.io/projected/724b46cc-9941-422b-b2c8-05ed9414a031-kube-api-access-vd6zj\") pod \"console-665655dc88-52ktb\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.588761 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.588655 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:27.882433 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:27.882355 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-665655dc88-52ktb"] Sep 10 16:18:27.885621 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:27.885589 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724b46cc_9941_422b_b2c8_05ed9414a031.slice/crio-fc2ca665c0d7d89b8826b3730a3510ee415ebed805e08d8ca90d1e453ca08d61 WatchSource:0}: Error finding container fc2ca665c0d7d89b8826b3730a3510ee415ebed805e08d8ca90d1e453ca08d61: Status 404 returned error can't find the container with id fc2ca665c0d7d89b8826b3730a3510ee415ebed805e08d8ca90d1e453ca08d61 Sep 10 16:18:28.006416 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:28.006378 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665655dc88-52ktb" event={"ID":"724b46cc-9941-422b-b2c8-05ed9414a031","Type":"ContainerStarted","Data":"fc2ca665c0d7d89b8826b3730a3510ee415ebed805e08d8ca90d1e453ca08d61"} Sep 10 16:18:28.915982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:28.915951 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h2jpq" Sep 10 16:18:29.062713 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:29.062656 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:29.162950 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:29.162916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:29.670309 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:29.670266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:29.744906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:29.744873 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:29.842820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:29.842782 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:29.941006 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:29.940902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:30.094458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.094419 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:30.140287 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.140255 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-4hsv4"] Sep 10 16:18:30.142356 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.142333 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.146721 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.146699 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 16:18:30.146841 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.146699 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 16:18:30.146945 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.146927 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 16:18:30.147264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.147007 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 16:18:30.147264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.147022 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 16:18:30.147264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.147008 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-52vzp\"" Sep 10 16:18:30.156964 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.156937 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-4hsv4"] Sep 10 16:18:30.170137 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.170103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:30.219963 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.219871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d197fbe3-3456-40ab-bad7-20390790af34-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.220167 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.220010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqb9z\" (UniqueName: \"kubernetes.io/projected/d197fbe3-3456-40ab-bad7-20390790af34-kube-api-access-cqb9z\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.220167 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.220089 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d197fbe3-3456-40ab-bad7-20390790af34-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.220167 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.220125 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d197fbe3-3456-40ab-bad7-20390790af34-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.242621 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.242585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:30.321262 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.321219 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d197fbe3-3456-40ab-bad7-20390790af34-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.321465 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.321280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d197fbe3-3456-40ab-bad7-20390790af34-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.321465 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.321339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d197fbe3-3456-40ab-bad7-20390790af34-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.321465 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.321416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqb9z\" (UniqueName: \"kubernetes.io/projected/d197fbe3-3456-40ab-bad7-20390790af34-kube-api-access-cqb9z\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.322297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.322269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d197fbe3-3456-40ab-bad7-20390790af34-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.324177 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.324154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d197fbe3-3456-40ab-bad7-20390790af34-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.324279 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.324153 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d197fbe3-3456-40ab-bad7-20390790af34-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.330139 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.330113 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqb9z\" (UniqueName: \"kubernetes.io/projected/d197fbe3-3456-40ab-bad7-20390790af34-kube-api-access-cqb9z\") pod \"prometheus-operator-6f4946db56-4hsv4\" (UID: \"d197fbe3-3456-40ab-bad7-20390790af34\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.452179 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.452145 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" Sep 10 16:18:30.841857 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.841813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:30.903719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:30.903680 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:31.019746 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:31.019708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665655dc88-52ktb" event={"ID":"724b46cc-9941-422b-b2c8-05ed9414a031","Type":"ContainerStarted","Data":"ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584"} Sep 10 16:18:31.040963 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:31.040913 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-665655dc88-52ktb" podStartSLOduration=1.002738008 podStartE2EDuration="4.040897038s" totalImagesPullingTime="3.03815903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:31.039643324 +0000 UTC m=+61.765962237" watchObservedRunningTime="2026-09-10 16:18:31.040897038 +0000 UTC m=+61.767215967" Sep 10 16:18:31.164564 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:31.164488 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-4hsv4"] Sep 10 16:18:31.167780 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:31.167747 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd197fbe3_3456_40ab_bad7_20390790af34.slice/crio-c175cbdb3b5f9cdccac7824d1408b5d3d896d1b62b5fa035c1ac73b7169641a9 WatchSource:0}: Error finding container c175cbdb3b5f9cdccac7824d1408b5d3d896d1b62b5fa035c1ac73b7169641a9: Status 404 returned error can't find the container with id c175cbdb3b5f9cdccac7824d1408b5d3d896d1b62b5fa035c1ac73b7169641a9 Sep 10 16:18:32.024701 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:32.024656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" event={"ID":"d197fbe3-3456-40ab-bad7-20390790af34","Type":"ContainerStarted","Data":"c175cbdb3b5f9cdccac7824d1408b5d3d896d1b62b5fa035c1ac73b7169641a9"} Sep 10 16:18:32.742941 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:32.742843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:33.030482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:33.030441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" event={"ID":"d197fbe3-3456-40ab-bad7-20390790af34","Type":"ContainerStarted","Data":"f63850faa2bb31809593465678da475d4c2842949e06c2741ad82ecb5a378458"} Sep 10 16:18:33.030876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:33.030505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" event={"ID":"d197fbe3-3456-40ab-bad7-20390790af34","Type":"ContainerStarted","Data":"8659f0c75e4b85f52a026f6d2e8b0da147be01ec00de94943d403a60588dc894"} Sep 10 16:18:33.053291 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:33.053237 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-4hsv4" podStartSLOduration=1.7539886770000002 podStartE2EDuration="3.053220705s" totalImagesPullingTime="1.299232028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:33.052923171 +0000 UTC m=+63.779242101" watchObservedRunningTime="2026-09-10 16:18:33.053220705 +0000 UTC m=+63.779539665" Sep 10 16:18:33.997772 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:33.997740 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sb6w5" Sep 10 16:18:34.038476 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.038438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.139250 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.139210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.256326 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.256251 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.370908 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.370872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.480998 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.480960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.520956 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.520867 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb"] Sep 10 16:18:34.541569 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.541533 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wvwvx"] Sep 10 16:18:34.541736 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.541651 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.544804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.544774 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 16:18:34.544954 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.544853 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kdqwx\"" Sep 10 16:18:34.545066 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.544784 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 16:18:34.555618 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.555595 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb"] Sep 10 16:18:34.555740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.555657 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j"] Sep 10 16:18:34.555802 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.555778 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.558695 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.558674 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 16:18:34.558804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.558787 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gmj9w\"" Sep 10 16:18:34.558983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.558968 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 16:18:34.559142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.559123 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 16:18:34.571822 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.571797 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j"] Sep 10 16:18:34.572000 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.571982 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.574886 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.574863 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 16:18:34.574886 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.574876 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-flz72\"" Sep 10 16:18:34.574886 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.574882 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 16:18:34.575150 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.574979 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 16:18:34.583382 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.583360 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.655393 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.655574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.655574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655436 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m29p\" (UniqueName: \"kubernetes.io/projected/fa17ef87-754e-433c-b22d-63e408216cb1-kube-api-access-7m29p\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.655574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655456 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.655574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bfdeec23-60c7-42ec-9725-940b1e1910ee-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.655574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.655574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655507 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-sys\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655601 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-textfile\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655651 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa17ef87-754e-433c-b22d-63e408216cb1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655679 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92ksx\" (UniqueName: \"kubernetes.io/projected/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-api-access-92ksx\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7377ae4f-ad20-483a-98c8-34f6e4d55eca-metrics-client-ca\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n779q\" (UniqueName: \"kubernetes.io/projected/7377ae4f-ad20-483a-98c8-34f6e4d55eca-kube-api-access-n779q\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-wtmp\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.655775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-accelerators-collector-config\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.656087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfdeec23-60c7-42ec-9725-940b1e1910ee-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.656087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-tls\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.656087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.655820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-root\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.676272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.676229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.756528 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-wtmp\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.756705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-accelerators-collector-config\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.756705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756558 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfdeec23-60c7-42ec-9725-940b1e1910ee-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.756705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-tls\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.756879 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-root\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.756879 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756669 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-wtmp\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.756879 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.756879 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756805 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-root\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.756879 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.757145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m29p\" (UniqueName: \"kubernetes.io/projected/fa17ef87-754e-433c-b22d-63e408216cb1-kube-api-access-7m29p\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.757145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.757145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bfdeec23-60c7-42ec-9725-940b1e1910ee-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.757145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.757145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.756993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.757145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757024 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-sys\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.757420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-textfile\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.757420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.757420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757216 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa17ef87-754e-433c-b22d-63e408216cb1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.757420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-accelerators-collector-config\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.757420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92ksx\" (UniqueName: \"kubernetes.io/projected/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-api-access-92ksx\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.757420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfdeec23-60c7-42ec-9725-940b1e1910ee-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.757738 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757550 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.757738 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:34.757683 2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 10 16:18:34.757919 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:34.757746 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-tls podName:fa17ef87-754e-433c-b22d-63e408216cb1 nodeName:}" failed. No retries permitted until 2026-09-10 16:18:35.25772743 +0000 UTC m=+65.984046345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-4p5cb" (UID: "fa17ef87-754e-433c-b22d-63e408216cb1") : secret "openshift-state-metrics-tls" not found Sep 10 16:18:34.757919 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7377ae4f-ad20-483a-98c8-34f6e4d55eca-sys\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.757919 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7377ae4f-ad20-483a-98c8-34f6e4d55eca-metrics-client-ca\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.758204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.757935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n779q\" (UniqueName: \"kubernetes.io/projected/7377ae4f-ad20-483a-98c8-34f6e4d55eca-kube-api-access-n779q\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.758561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.758475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bfdeec23-60c7-42ec-9725-940b1e1910ee-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.758561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.758506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7377ae4f-ad20-483a-98c8-34f6e4d55eca-metrics-client-ca\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.758561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.758513 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-textfile\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.758775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.758643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa17ef87-754e-433c-b22d-63e408216cb1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.759657 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.759618 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.759788 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.759768 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-tls\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.759849 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.759785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.760072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.760030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7377ae4f-ad20-483a-98c8-34f6e4d55eca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.760277 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.760259 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.768587 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.768557 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.769212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.769192 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m29p\" (UniqueName: \"kubernetes.io/projected/fa17ef87-754e-433c-b22d-63e408216cb1-kube-api-access-7m29p\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:34.769302 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.769225 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n779q\" (UniqueName: \"kubernetes.io/projected/7377ae4f-ad20-483a-98c8-34f6e4d55eca-kube-api-access-n779q\") pod \"node-exporter-wvwvx\" (UID: \"7377ae4f-ad20-483a-98c8-34f6e4d55eca\") " pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.769503 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.769484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92ksx\" (UniqueName: \"kubernetes.io/projected/bfdeec23-60c7-42ec-9725-940b1e1910ee-kube-api-access-92ksx\") pod \"kube-state-metrics-74cb57df7-d6m2j\" (UID: \"bfdeec23-60c7-42ec-9725-940b1e1910ee\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.828156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.828023 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:34.864321 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.864283 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wvwvx" Sep 10 16:18:34.881183 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:34.881149 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" Sep 10 16:18:34.903721 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:34.903682 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7377ae4f_ad20_483a_98c8_34f6e4d55eca.slice/crio-3be6f65ae0ad7dc07ccd99bec8fab39aef6e8138619cbaa6561102b4df3915c4 WatchSource:0}: Error finding container 3be6f65ae0ad7dc07ccd99bec8fab39aef6e8138619cbaa6561102b4df3915c4: Status 404 returned error can't find the container with id 3be6f65ae0ad7dc07ccd99bec8fab39aef6e8138619cbaa6561102b4df3915c4 Sep 10 16:18:35.023694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.023654 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:35.038444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.038414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wvwvx" event={"ID":"7377ae4f-ad20-483a-98c8-34f6e4d55eca","Type":"ContainerStarted","Data":"3be6f65ae0ad7dc07ccd99bec8fab39aef6e8138619cbaa6561102b4df3915c4"} Sep 10 16:18:35.121077 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.120984 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:35.211029 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.211007 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j"] Sep 10 16:18:35.213538 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:35.213508 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdeec23_60c7_42ec_9725_940b1e1910ee.slice/crio-028fda73b3f4f1a3f2c99aa9f753691bc4ad67ba6ad22794e8001c4e51d6ad60 WatchSource:0}: Error finding container 028fda73b3f4f1a3f2c99aa9f753691bc4ad67ba6ad22794e8001c4e51d6ad60: Status 404 returned error can't find the container with id 028fda73b3f4f1a3f2c99aa9f753691bc4ad67ba6ad22794e8001c4e51d6ad60 Sep 10 16:18:35.227274 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.227248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:35.262676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.262633 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:35.265107 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.265085 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa17ef87-754e-433c-b22d-63e408216cb1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-4p5cb\" (UID: \"fa17ef87-754e-433c-b22d-63e408216cb1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:35.452305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.452216 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" Sep 10 16:18:35.463935 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.463902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:35.464079 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.463978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:35.466840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.466810 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 16:18:35.466840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.466826 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 16:18:35.476884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.476793 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944b5648-71ef-4980-927f-351334db589e-metrics-certs\") pod \"network-metrics-daemon-hn92b\" (UID: \"944b5648-71ef-4980-927f-351334db589e\") " pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:35.478460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.478327 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb5f5c7-9cca-44c3-9013-6fcc3788f877-original-pull-secret\") pod \"global-pull-secret-syncer-l849b\" (UID: \"3cb5f5c7-9cca-44c3-9013-6fcc3788f877\") " pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:35.490923 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.490888 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:35.564862 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.564825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:35.567979 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.567941 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 16:18:35.576229 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.576199 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:35.577901 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.577880 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 16:18:35.581882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.581855 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:18:35.584811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.584787 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.588292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.588271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xwl\" (UniqueName: \"kubernetes.io/projected/ed0d041c-4673-444b-8392-02de7cc0958f-kube-api-access-98xwl\") pod \"network-check-target-mvhbl\" (UID: \"ed0d041c-4673-444b-8392-02de7cc0958f\") " pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:35.590759 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.590736 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nsnhw\"" Sep 10 16:18:35.591062 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.591028 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 16:18:35.591210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.591163 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 16:18:35.591299 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.591203 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 16:18:35.592022 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.592004 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 16:18:35.592249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.592234 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 16:18:35.592424 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.592404 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 16:18:35.593090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.592731 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 16:18:35.593090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.592920 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 16:18:35.593599 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.593181 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 16:18:35.633136 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.633099 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:18:35.662954 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.662919 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l849b" Sep 10 16:18:35.666044 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.665999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666167 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-web-config\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666167 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666100 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-out\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666252 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666252 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmw2\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-kube-api-access-rpmw2\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666326 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666257 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666326 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666411 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666445 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666493 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666458 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666493 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666488 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666600 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.666600 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.666543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.672826 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.672804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j8r7d\"" Sep 10 16:18:35.676436 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.676398 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htvxx\"" Sep 10 16:18:35.679346 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.679326 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn92b" Sep 10 16:18:35.683822 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.683792 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:35.742217 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.742187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:35.744158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.743247 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb"] Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.767825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.767868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.767904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.767930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.767959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.767983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.768007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.768065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.768090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-web-config\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.768140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-out\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.768165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.768189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmw2\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-kube-api-access-rpmw2\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.768238 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:35.768636 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle podName:9a5bf842-e92d-4b52-a664-6de26c861d6f nodeName:}" failed. No retries permitted until 2026-09-10 16:18:36.268610267 +0000 UTC m=+66.994929177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f") : configmap references non-existent config key: ca-bundle.crt Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:35.768989 2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 10 16:18:35.769864 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:35.769086 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls podName:9a5bf842-e92d-4b52-a664-6de26c861d6f nodeName:}" failed. No retries permitted until 2026-09-10 16:18:36.26902732 +0000 UTC m=+66.995346242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f") : secret "alertmanager-main-tls" not found Sep 10 16:18:35.771340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.771316 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.772270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.771931 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.774787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.774740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-web-config\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.775409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.775347 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.775972 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.775858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.775972 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.775922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-out\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.776614 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.776381 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.776786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.776762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.779274 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.779224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.781949 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.781903 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.784663 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.784405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmw2\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-kube-api-access-rpmw2\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:35.806977 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.806891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:35.944780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.944744 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l849b"] Sep 10 16:18:35.983526 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.983422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hn92b"] Sep 10 16:18:35.985484 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:35.985323 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mvhbl"] Sep 10 16:18:36.044789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.044742 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" event={"ID":"bfdeec23-60c7-42ec-9725-940b1e1910ee","Type":"ContainerStarted","Data":"028fda73b3f4f1a3f2c99aa9f753691bc4ad67ba6ad22794e8001c4e51d6ad60"} Sep 10 16:18:36.052943 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.052908 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" event={"ID":"fa17ef87-754e-433c-b22d-63e408216cb1","Type":"ContainerStarted","Data":"2238136dbdae2578b30d0a4a9b2a0d89373bd98471ee037d5496bb2ecef16210"} Sep 10 16:18:36.053138 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.053058 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" event={"ID":"fa17ef87-754e-433c-b22d-63e408216cb1","Type":"ContainerStarted","Data":"3150cc0a4435d2aef3106e9e38c6b73ed980eb74319ac065fc87b507f94f6031"} Sep 10 16:18:36.077228 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:36.077192 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944b5648_71ef_4980_927f_351334db589e.slice/crio-290061fdc0d6a357e0d6b883e6ac790ade3d00d5fc5c0e99dc884e39cb4c0c19 WatchSource:0}: Error finding container 290061fdc0d6a357e0d6b883e6ac790ade3d00d5fc5c0e99dc884e39cb4c0c19: Status 404 returned error can't find the container with id 290061fdc0d6a357e0d6b883e6ac790ade3d00d5fc5c0e99dc884e39cb4c0c19 Sep 10 16:18:36.080707 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:36.080615 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0d041c_4673_444b_8392_02de7cc0958f.slice/crio-a791865e20fd4ef0bfe99143c04a1d02bf23dd65bb1163ebf54bc654a42a0399 WatchSource:0}: Error finding container a791865e20fd4ef0bfe99143c04a1d02bf23dd65bb1163ebf54bc654a42a0399: Status 404 returned error can't find the container with id a791865e20fd4ef0bfe99143c04a1d02bf23dd65bb1163ebf54bc654a42a0399 Sep 10 16:18:36.103882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.103859 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:36.273297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.273254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:36.273677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.273309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:36.274132 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.274104 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:36.275648 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.275618 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:36.495769 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.495732 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:18:36.860341 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:36.860234 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:18:37.063625 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.062527 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" event={"ID":"bfdeec23-60c7-42ec-9725-940b1e1910ee","Type":"ContainerStarted","Data":"e0a45798f693fed2eb5596a8ac278213fe8a723a9607757fa73252082211a318"} Sep 10 16:18:37.063625 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.062615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" event={"ID":"bfdeec23-60c7-42ec-9725-940b1e1910ee","Type":"ContainerStarted","Data":"56e9dabdacefc70ddfa4d07efd4d744c546a23c162d5b9ecbb471e1eadb9dbf9"} Sep 10 16:18:37.063625 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.062647 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" event={"ID":"bfdeec23-60c7-42ec-9725-940b1e1910ee","Type":"ContainerStarted","Data":"12c542e8b2b626fc6461a9ca1e51a858e661c500a9725bbe33178d8fbeb9005f"} Sep 10 16:18:37.065449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.065405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerStarted","Data":"44f35306976d598353909b060fc350b482b474d79d80b1bbe267940ba228998c"} Sep 10 16:18:37.067726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.067649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn92b" event={"ID":"944b5648-71ef-4980-927f-351334db589e","Type":"ContainerStarted","Data":"290061fdc0d6a357e0d6b883e6ac790ade3d00d5fc5c0e99dc884e39cb4c0c19"} Sep 10 16:18:37.070506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.070467 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l849b" event={"ID":"3cb5f5c7-9cca-44c3-9013-6fcc3788f877","Type":"ContainerStarted","Data":"b37bd6d3716d8ce3d5b067138f746cc3a4f0762abf292462cff9942d3e6de35e"} Sep 10 16:18:37.073066 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.072710 2562 generic.go:356] "Generic (PLEG): container finished" podID="7377ae4f-ad20-483a-98c8-34f6e4d55eca" containerID="bbfb30ef75f71710ae9943a33f71c99dca3a80786ccb9759c608213998bec7c4" exitCode=0 Sep 10 16:18:37.073066 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.072789 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wvwvx" event={"ID":"7377ae4f-ad20-483a-98c8-34f6e4d55eca","Type":"ContainerDied","Data":"bbfb30ef75f71710ae9943a33f71c99dca3a80786ccb9759c608213998bec7c4"} Sep 10 16:18:37.076134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.076027 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" event={"ID":"fa17ef87-754e-433c-b22d-63e408216cb1","Type":"ContainerStarted","Data":"a92909407f9348c3445f2fe131d77fc3c9b8fcea75bde4e352b96c8baf8f6e7b"} Sep 10 16:18:37.078174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.078118 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mvhbl" event={"ID":"ed0d041c-4673-444b-8392-02de7cc0958f","Type":"ContainerStarted","Data":"a791865e20fd4ef0bfe99143c04a1d02bf23dd65bb1163ebf54bc654a42a0399"} Sep 10 16:18:37.097421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.096560 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-d6m2j" podStartSLOduration=1.7132027 podStartE2EDuration="3.096539625s" totalImagesPullingTime="1.383336925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:37.093636214 +0000 UTC m=+67.819955176" watchObservedRunningTime="2026-09-10 16:18:37.096539625 +0000 UTC m=+67.822858558" Sep 10 16:18:37.444289 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.442915 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-745985f847-r7tgz"] Sep 10 16:18:37.446068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.445532 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.458533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.457228 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745985f847-r7tgz"] Sep 10 16:18:37.460369 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.460249 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 16:18:37.461601 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.461537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:37.483658 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.483491 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-oauth-config\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.483658 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.483540 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-console-config\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.483658 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.483590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-oauth-serving-cert\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.483938 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.483699 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-trusted-ca-bundle\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.483938 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.483750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8nks\" (UniqueName: \"kubernetes.io/projected/1df1be45-a88c-4a16-af73-2c4680432842-kube-api-access-l8nks\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.483938 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.483823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-service-ca\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.483938 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.483873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-serving-cert\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.558961 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.558923 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:37.584992 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.584959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-service-ca\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.585171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.585005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-serving-cert\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.585171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.585053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-oauth-config\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.585171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.585083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-console-config\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.585171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.585125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-oauth-serving-cert\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.585374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.585161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-trusted-ca-bundle\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.585374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.585230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8nks\" (UniqueName: \"kubernetes.io/projected/1df1be45-a88c-4a16-af73-2c4680432842-kube-api-access-l8nks\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.585839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.585812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-service-ca\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.587559 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.586112 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-console-config\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.587559 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.586594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-oauth-serving-cert\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.587559 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.587512 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-trusted-ca-bundle\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.589739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.589138 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:37.589739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.589197 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:37.590463 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.590414 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-oauth-config\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.591985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.591938 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-serving-cert\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.595144 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.595100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8nks\" (UniqueName: \"kubernetes.io/projected/1df1be45-a88c-4a16-af73-2c4680432842-kube-api-access-l8nks\") pod \"console-745985f847-r7tgz\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.595533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.595508 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:37.763393 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.763360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:37.771440 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.771391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:37.861835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:37.861731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:38.027819 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.027721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:38.086093 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.086057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wvwvx" event={"ID":"7377ae4f-ad20-483a-98c8-34f6e4d55eca","Type":"ContainerStarted","Data":"d289069b26a61c2d93229fdc0a213069cd765064a971707bc360638324932981"} Sep 10 16:18:38.086276 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.086123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wvwvx" event={"ID":"7377ae4f-ad20-483a-98c8-34f6e4d55eca","Type":"ContainerStarted","Data":"3248a3a5e343ea5a3af70ec7a22040484abcd01e7679eec1483e022a298de6da"} Sep 10 16:18:38.092230 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.092170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:18:38.110212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.109265 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wvwvx" podStartSLOduration=2.8378330370000002 podStartE2EDuration="4.109243413s" totalImagesPullingTime="1.201797104s" totalInitContainerRuntime="69.613272ms" isStatefulPod=true podCreationTimestamp="2026-09-10 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:38.108419292 +0000 UTC m=+68.834738233" watchObservedRunningTime="2026-09-10 16:18:38.109243413 +0000 UTC m=+68.835562347" Sep 10 16:18:38.435987 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.435909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:38.510655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.510582 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745985f847-r7tgz"] Sep 10 16:18:38.515548 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:38.515513 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df1be45_a88c_4a16_af73_2c4680432842.slice/crio-0237ff0a5e4ec5c2bcfa31b85ddac88ba07c2d6fa3d5176f3f04c5c00434beb7 WatchSource:0}: Error finding container 0237ff0a5e4ec5c2bcfa31b85ddac88ba07c2d6fa3d5176f3f04c5c00434beb7: Status 404 returned error can't find the container with id 0237ff0a5e4ec5c2bcfa31b85ddac88ba07c2d6fa3d5176f3f04c5c00434beb7 Sep 10 16:18:38.592618 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.592590 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-546cf76c59-9w54t"] Sep 10 16:18:38.595458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.595377 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.598312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.598289 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 16:18:38.599850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.598745 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kd2c4\"" Sep 10 16:18:38.599850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.599026 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-949tr43gdt0rb\"" Sep 10 16:18:38.599850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.599222 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 16:18:38.599850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.599391 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 16:18:38.599850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.599545 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 16:18:38.599850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.599708 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 16:18:38.609983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.609885 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-546cf76c59-9w54t"] Sep 10 16:18:38.651398 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.651362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:38.696192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.696152 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-tls\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.696355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.696201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.696355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.696236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.696355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.696276 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-grpc-tls\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.696355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.696298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-metrics-client-ca\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.696355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.696331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.696668 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.696359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.696668 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.696416 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52lj\" (UniqueName: \"kubernetes.io/projected/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-kube-api-access-g52lj\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.712892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.712831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:38.797198 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.797114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-tls\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.797198 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.797166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.797198 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.797187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.797444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.797221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-grpc-tls\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.797444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.797248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-metrics-client-ca\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.797444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.797282 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.797444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.797311 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.797444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.797370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52lj\" (UniqueName: \"kubernetes.io/projected/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-kube-api-access-g52lj\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.798575 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.798515 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-metrics-client-ca\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.800788 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.800731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.800916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.800815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-grpc-tls\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.801206 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.801177 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.801373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.801351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.801474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.801449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.802217 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.802163 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-secret-thanos-querier-tls\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.806323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.806297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52lj\" (UniqueName: \"kubernetes.io/projected/a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad-kube-api-access-g52lj\") pod \"thanos-querier-546cf76c59-9w54t\" (UID: \"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad\") " pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:38.819578 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.819544 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69f846fc4-5gghq"] Sep 10 16:18:38.822366 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.822345 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:38.825305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.825279 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8c2p4d8gvlp3\"" Sep 10 16:18:38.825305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.825291 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 16:18:38.825551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.825537 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 16:18:38.825591 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.825566 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 16:18:38.825634 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.825619 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 16:18:38.825670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.825537 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cl62k\"" Sep 10 16:18:38.831793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.831775 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69f846fc4-5gghq"] Sep 10 16:18:38.898361 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.898321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-client-ca-bundle\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:38.898537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.898382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031e2916-b4b3-474e-9041-22ed51454669-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:38.898537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.898425 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/031e2916-b4b3-474e-9041-22ed51454669-audit-log\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:38.898537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.898463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-secret-metrics-server-tls\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:38.898537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.898498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczcn\" (UniqueName: \"kubernetes.io/projected/031e2916-b4b3-474e-9041-22ed51454669-kube-api-access-qczcn\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:38.898537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.898531 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/031e2916-b4b3-474e-9041-22ed51454669-metrics-server-audit-profiles\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:38.898794 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.898567 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-secret-metrics-server-client-certs\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:38.909319 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.909283 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:39.000064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.999852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031e2916-b4b3-474e-9041-22ed51454669-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.000064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.999914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/031e2916-b4b3-474e-9041-22ed51454669-audit-log\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.000064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:38.999960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-secret-metrics-server-tls\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.000064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.000001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczcn\" (UniqueName: \"kubernetes.io/projected/031e2916-b4b3-474e-9041-22ed51454669-kube-api-access-qczcn\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.000064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.000069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/031e2916-b4b3-474e-9041-22ed51454669-metrics-server-audit-profiles\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.000428 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.000108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-secret-metrics-server-client-certs\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.000428 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.000174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-client-ca-bundle\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.001574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.000675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031e2916-b4b3-474e-9041-22ed51454669-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.001574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.000805 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/031e2916-b4b3-474e-9041-22ed51454669-audit-log\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.001574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.001503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/031e2916-b4b3-474e-9041-22ed51454669-metrics-server-audit-profiles\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.003899 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.003427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-client-ca-bundle\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.003899 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.003869 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-secret-metrics-server-tls\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.004382 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.004360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/031e2916-b4b3-474e-9041-22ed51454669-secret-metrics-server-client-certs\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.009545 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.009520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczcn\" (UniqueName: \"kubernetes.io/projected/031e2916-b4b3-474e-9041-22ed51454669-kube-api-access-qczcn\") pod \"metrics-server-69f846fc4-5gghq\" (UID: \"031e2916-b4b3-474e-9041-22ed51454669\") " pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.095730 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.095634 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerID="66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80" exitCode=0 Sep 10 16:18:39.095730 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.095713 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerDied","Data":"66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80"} Sep 10 16:18:39.098073 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.098028 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn92b" event={"ID":"944b5648-71ef-4980-927f-351334db589e","Type":"ContainerStarted","Data":"91b212d9c127ef4d73ea9a4364be1d27c6bc1fe8fcaf3c36286c006706f3ea85"} Sep 10 16:18:39.098242 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.098224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn92b" event={"ID":"944b5648-71ef-4980-927f-351334db589e","Type":"ContainerStarted","Data":"87db1b7a46bf9c0647722c9f61590e6acaa2d56e0bb754d56211614438611945"} Sep 10 16:18:39.100758 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.100734 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" event={"ID":"fa17ef87-754e-433c-b22d-63e408216cb1","Type":"ContainerStarted","Data":"351e0cffbb709cfae65de96ece66bb1393450215b1de5706e099dabf4f236429"} Sep 10 16:18:39.102707 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.102669 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745985f847-r7tgz" event={"ID":"1df1be45-a88c-4a16-af73-2c4680432842","Type":"ContainerStarted","Data":"5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941"} Sep 10 16:18:39.102794 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.102711 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745985f847-r7tgz" event={"ID":"1df1be45-a88c-4a16-af73-2c4680432842","Type":"ContainerStarted","Data":"0237ff0a5e4ec5c2bcfa31b85ddac88ba07c2d6fa3d5176f3f04c5c00434beb7"} Sep 10 16:18:39.132361 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.132326 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:39.142740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.142674 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-4p5cb" podStartSLOduration=3.073948617 podStartE2EDuration="5.142653603s" totalImagesPullingTime="2.068704986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:39.141710486 +0000 UTC m=+69.868029440" watchObservedRunningTime="2026-09-10 16:18:39.142653603 +0000 UTC m=+69.868972533" Sep 10 16:18:39.166868 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.166802 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hn92b" podStartSLOduration=67.023429801 podStartE2EDuration="1m9.166778526s" totalImagesPullingTime="2.143348725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:39.165867121 +0000 UTC m=+69.892186051" watchObservedRunningTime="2026-09-10 16:18:39.166778526 +0000 UTC m=+69.893097457" Sep 10 16:18:39.194858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.194772 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-745985f847-r7tgz" podStartSLOduration=2.194754812 podStartE2EDuration="2.194754812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:39.190679201 +0000 UTC m=+69.916998133" watchObservedRunningTime="2026-09-10 16:18:39.194754812 +0000 UTC m=+69.921073753" Sep 10 16:18:39.209143 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.209117 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-546cf76c59-9w54t"] Sep 10 16:18:39.211629 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:39.211567 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05dc8a2_cb9e_4932_b7da_b71c3ee8fcad.slice/crio-81458a1cbb4eeb264acaf5bbb9e3b77791461df52c04250bebaafcc6ac148ddd WatchSource:0}: Error finding container 81458a1cbb4eeb264acaf5bbb9e3b77791461df52c04250bebaafcc6ac148ddd: Status 404 returned error can't find the container with id 81458a1cbb4eeb264acaf5bbb9e3b77791461df52c04250bebaafcc6ac148ddd Sep 10 16:18:39.302343 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.302297 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:39.318583 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.318549 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h"] Sep 10 16:18:39.322051 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.322011 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:39.325814 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.325642 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 16:18:39.325814 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.325758 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-s6wjz\"" Sep 10 16:18:39.333284 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.332612 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h"] Sep 10 16:18:39.372767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.372655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:39.404749 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.404707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-9c79h\" (UID: \"a7b17f6e-bdf2-4093-9e9a-7e76453423ef\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:39.404910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.404803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wl2\" (UniqueName: \"kubernetes.io/projected/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-kube-api-access-h2wl2\") pod \"monitoring-plugin-85bbbbbf4-9c79h\" (UID: \"a7b17f6e-bdf2-4093-9e9a-7e76453423ef\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:39.416742 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.416711 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69f846fc4-5gghq"] Sep 10 16:18:39.505817 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.505779 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-9c79h\" (UID: \"a7b17f6e-bdf2-4093-9e9a-7e76453423ef\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:39.506001 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.505861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wl2\" (UniqueName: \"kubernetes.io/projected/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-kube-api-access-h2wl2\") pod \"monitoring-plugin-85bbbbbf4-9c79h\" (UID: \"a7b17f6e-bdf2-4093-9e9a-7e76453423ef\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:39.506207 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:39.506185 2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 10 16:18:39.506314 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:18:39.506282 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-monitoring-plugin-cert podName:a7b17f6e-bdf2-4093-9e9a-7e76453423ef nodeName:}" failed. No retries permitted until 2026-09-10 16:18:40.006255971 +0000 UTC m=+70.732574896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-9c79h" (UID: "a7b17f6e-bdf2-4093-9e9a-7e76453423ef") : secret "monitoring-plugin-cert" not found Sep 10 16:18:39.515408 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.515379 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wl2\" (UniqueName: \"kubernetes.io/projected/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-kube-api-access-h2wl2\") pod \"monitoring-plugin-85bbbbbf4-9c79h\" (UID: \"a7b17f6e-bdf2-4093-9e9a-7e76453423ef\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:39.564573 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.564470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:39.842633 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.842430 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7748f44994-6bksq"] Sep 10 16:18:39.845389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.845363 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:39.848439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.848412 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g9hct\"" Sep 10 16:18:39.848559 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.848500 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 16:18:39.848618 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.848606 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 16:18:39.848819 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.848804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 16:18:39.848876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.848840 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 16:18:39.848920 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.848840 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 16:18:39.854804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.854784 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 16:18:39.860583 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.860553 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7748f44994-6bksq"] Sep 10 16:18:39.909259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.909222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsqw\" (UniqueName: \"kubernetes.io/projected/5f816f97-e930-4fb4-b514-d070fc2b1686-kube-api-access-4wsqw\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:39.909444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.909268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-secret-telemeter-client\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:39.909444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.909302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-telemeter-client-tls\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:39.909444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.909378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:39.909444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.909411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-federate-client-tls\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:39.909613 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.909460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-metrics-client-ca\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:39.909613 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.909493 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:39.909613 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:39.909512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-serving-certs-ca-bundle\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.010552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.010732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-federate-client-tls\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.010732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010620 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-9c79h\" (UID: \"a7b17f6e-bdf2-4093-9e9a-7e76453423ef\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:40.010732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-metrics-client-ca\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.010732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.010732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-serving-certs-ca-bundle\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.010997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsqw\" (UniqueName: \"kubernetes.io/projected/5f816f97-e930-4fb4-b514-d070fc2b1686-kube-api-access-4wsqw\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.010997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-secret-telemeter-client\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.010997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.010793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-telemeter-client-tls\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.011909 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.011829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-serving-certs-ca-bundle\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.011909 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.011851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-metrics-client-ca\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.012185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.012159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f816f97-e930-4fb4-b514-d070fc2b1686-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.014013 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.013983 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-telemeter-client-tls\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.014288 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.014261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-secret-telemeter-client\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.014390 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.014361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-federate-client-tls\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.014452 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.014408 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f816f97-e930-4fb4-b514-d070fc2b1686-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.014546 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.014526 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7b17f6e-bdf2-4093-9e9a-7e76453423ef-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-9c79h\" (UID: \"a7b17f6e-bdf2-4093-9e9a-7e76453423ef\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:40.021199 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.021174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsqw\" (UniqueName: \"kubernetes.io/projected/5f816f97-e930-4fb4-b514-d070fc2b1686-kube-api-access-4wsqw\") pod \"telemeter-client-7748f44994-6bksq\" (UID: \"5f816f97-e930-4fb4-b514-d070fc2b1686\") " pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.109203 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.109104 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" event={"ID":"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad","Type":"ContainerStarted","Data":"81458a1cbb4eeb264acaf5bbb9e3b77791461df52c04250bebaafcc6ac148ddd"} Sep 10 16:18:40.157699 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.157657 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" Sep 10 16:18:40.237802 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.237763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:40.752421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.752328 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:40.758586 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.758552 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:18:40.762784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.762751 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.766407 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.766377 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 16:18:40.766711 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.766692 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 16:18:40.766889 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.766874 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8t5desa4u6c0k\"" Sep 10 16:18:40.768094 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.767891 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 16:18:40.768094 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.767945 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 16:18:40.768094 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.767979 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bzbn4\"" Sep 10 16:18:40.768324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768127 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 16:18:40.768324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768204 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 16:18:40.768324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768219 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 16:18:40.768324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768305 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 16:18:40.768517 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768339 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 16:18:40.768517 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768221 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 16:18:40.768517 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768482 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 16:18:40.768771 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768756 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 16:18:40.768877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.768862 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 16:18:40.771141 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.771119 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 16:18:40.780182 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.780097 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:18:40.821101 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-config\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821211 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821240 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbr6b\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-kube-api-access-wbr6b\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-config-out\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821399 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821486 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-web-config\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821613 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.821958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.821642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbr6b\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-kube-api-access-wbr6b\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922364 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-config-out\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-web-config\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922967 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922856 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922967 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.922967 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.923144 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.922982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.923144 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.923007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.923144 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.923032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.923144 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.923104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-config\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.923571 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.923456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.924576 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.924516 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.924686 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.924646 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.926064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.925249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.926064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.925533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.926064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.925961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.926590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.926364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-web-config\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.927108 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.927080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.927286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.927260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-config-out\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.927825 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.927801 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.929786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.929398 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.929786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.929553 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.929786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.929748 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.930443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.930419 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.930628 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.930574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.930728 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.930672 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-config\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.931787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.931743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.932056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.932023 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:40.937450 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:40.937424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbr6b\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-kube-api-access-wbr6b\") pod \"prometheus-k8s-0\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:41.078174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:41.078079 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:41.126418 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:41.126388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:41.186772 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:41.186685 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:41.815090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:41.815052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:41.915113 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:41.915069 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:41.968566 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:41.968533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:42.037751 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:42.037714 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031e2916_b4b3_474e_9041_22ed51454669.slice/crio-5f68af441c395e67a10ab2549010a9c332dac631c4fa9efbc50f4c558097165d WatchSource:0}: Error finding container 5f68af441c395e67a10ab2549010a9c332dac631c4fa9efbc50f4c558097165d: Status 404 returned error can't find the container with id 5f68af441c395e67a10ab2549010a9c332dac631c4fa9efbc50f4c558097165d Sep 10 16:18:42.121127 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:42.121008 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" event={"ID":"031e2916-b4b3-474e-9041-22ed51454669","Type":"ContainerStarted","Data":"5f68af441c395e67a10ab2549010a9c332dac631c4fa9efbc50f4c558097165d"} Sep 10 16:18:42.731949 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:42.731882 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h"] Sep 10 16:18:42.737973 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:42.737936 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:18:42.739820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:42.739794 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7748f44994-6bksq"] Sep 10 16:18:42.739991 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:18:42.739834 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87c0e89_0ac4_4d97_96f4_030e30776c36.slice/crio-e3f2950c18e2131458736ff7b0b77ad8483102938bb6c3cd200610b57f69362f WatchSource:0}: Error finding container e3f2950c18e2131458736ff7b0b77ad8483102938bb6c3cd200610b57f69362f: Status 404 returned error can't find the container with id e3f2950c18e2131458736ff7b0b77ad8483102938bb6c3cd200610b57f69362f Sep 10 16:18:43.130371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.130327 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerStarted","Data":"bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd"} Sep 10 16:18:43.130793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.130412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerStarted","Data":"410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56"} Sep 10 16:18:43.130793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.130449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerStarted","Data":"464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5"} Sep 10 16:18:43.130793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.130474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerStarted","Data":"7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390"} Sep 10 16:18:43.130793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.130504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerStarted","Data":"d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716"} Sep 10 16:18:43.132632 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.132598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l849b" event={"ID":"3cb5f5c7-9cca-44c3-9013-6fcc3788f877","Type":"ContainerStarted","Data":"1a85ad623711c1c243b9886e267b7baaf83890b0173a05fc7f19b8207749bce7"} Sep 10 16:18:43.134379 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.134349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" event={"ID":"5f816f97-e930-4fb4-b514-d070fc2b1686","Type":"ContainerStarted","Data":"ea7fb26a01bd3090a198fbb0396bbc7807dce6f9d61cb98fdab8d56bec6656d7"} Sep 10 16:18:43.136329 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.136303 2562 generic.go:356] "Generic (PLEG): container finished" podID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerID="2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4" exitCode=0 Sep 10 16:18:43.136434 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.136396 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerDied","Data":"2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4"} Sep 10 16:18:43.136492 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.136480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerStarted","Data":"e3f2950c18e2131458736ff7b0b77ad8483102938bb6c3cd200610b57f69362f"} Sep 10 16:18:43.138228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.138206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mvhbl" event={"ID":"ed0d041c-4673-444b-8392-02de7cc0958f","Type":"ContainerStarted","Data":"9bf99d738003f61947431de0b0257c9c02dc9bf8731d1b611d9e21acfd3271d7"} Sep 10 16:18:43.138430 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.138359 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:18:43.139623 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.139599 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" event={"ID":"a7b17f6e-bdf2-4093-9e9a-7e76453423ef","Type":"ContainerStarted","Data":"5ae41db2a51695370ab1970e693e0e84012e10be7c3f13646b8bd3aebebf1132"} Sep 10 16:18:43.149975 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.149921 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l849b" podStartSLOduration=66.768914599 podStartE2EDuration="1m13.149902553s" totalImagesPullingTime="6.380987954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:43.148122802 +0000 UTC m=+73.874441735" watchObservedRunningTime="2026-09-10 16:18:43.149902553 +0000 UTC m=+73.876221484" Sep 10 16:18:43.195115 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.195059 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mvhbl" podStartSLOduration=66.843112369 podStartE2EDuration="1m13.195021553s" totalImagesPullingTime="6.351909184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:43.194404442 +0000 UTC m=+73.920723371" watchObservedRunningTime="2026-09-10 16:18:43.195021553 +0000 UTC m=+73.921340482" Sep 10 16:18:43.448430 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.448336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:43.454449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.453687 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-745985f847-r7tgz"] Sep 10 16:18:43.540856 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:43.540780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:45.168485 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:45.168413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:45.269708 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:45.269613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:45.444959 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:45.444924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:46.165273 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.165181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerStarted","Data":"3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c"} Sep 10 16:18:46.169068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.169019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" event={"ID":"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad","Type":"ContainerStarted","Data":"411db08135d4592a5a97aa327e40feaa79a72ecf5bb38e156a97f90257a9a51d"} Sep 10 16:18:46.169523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.169113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" event={"ID":"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad","Type":"ContainerStarted","Data":"56dbbdf2c5d8a078249c368e0d70801ce6346e60acf54f2392eaf6831e0f6c33"} Sep 10 16:18:46.169523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.169160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" event={"ID":"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad","Type":"ContainerStarted","Data":"3a27fda7296eb0540444ee737bf1cf5619531175fcf69de30af8f72637750889"} Sep 10 16:18:46.169523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.169198 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" event={"ID":"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad","Type":"ContainerStarted","Data":"ab9f9a80e40ba77c45d2199a747d23e22916bc2c7537df01d30f0ec6cac0860a"} Sep 10 16:18:46.169523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.169244 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:46.169523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.169284 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" event={"ID":"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad","Type":"ContainerStarted","Data":"d264ea9493ed5165027e632d809cba292df6e3f3b8a936145bdf8ae0e4acd876"} Sep 10 16:18:46.169523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.169360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" event={"ID":"a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad","Type":"ContainerStarted","Data":"ba5b1f925c76ed929f5b32bf90e98145b975853fb2dac6e4c6fcca2cce91244e"} Sep 10 16:18:46.171609 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.171571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" event={"ID":"5f816f97-e930-4fb4-b514-d070fc2b1686","Type":"ContainerStarted","Data":"0796cf9367d636c7f1cb5fdbcc7ea091876402696903f037dda417faefb6d4ca"} Sep 10 16:18:46.171735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.171632 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" event={"ID":"5f816f97-e930-4fb4-b514-d070fc2b1686","Type":"ContainerStarted","Data":"4f3755e0dbb5dec200b758feeeb9a3743091f19bc246f894f0e8dfc680c5feb8"} Sep 10 16:18:46.171735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.171653 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" event={"ID":"5f816f97-e930-4fb4-b514-d070fc2b1686","Type":"ContainerStarted","Data":"f1804dbef27e4a2258e86f0c18fd453ea750051329d80505d29d8d952e72f61c"} Sep 10 16:18:46.173943 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.173905 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" event={"ID":"031e2916-b4b3-474e-9041-22ed51454669","Type":"ContainerStarted","Data":"96a41ae583814b097f5436fade1f932970c6d578e5750851a72d1e2ab43f5fe8"} Sep 10 16:18:46.175740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.175714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" event={"ID":"a7b17f6e-bdf2-4093-9e9a-7e76453423ef","Type":"ContainerStarted","Data":"fb16574fb5b1318b87d3e8649261dfc0d44d845379cc8ff7e0383ca7c9757965"} Sep 10 16:18:46.175887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.175865 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:46.181882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.181860 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" Sep 10 16:18:46.197631 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.197191 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.977025952 podStartE2EDuration="11.197169792s" totalImagesPullingTime="7.194283558s" totalInitContainerRuntime="25.860282ms" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:46.196583934 +0000 UTC m=+76.922902865" watchObservedRunningTime="2026-09-10 16:18:46.197169792 +0000 UTC m=+76.923488724" Sep 10 16:18:46.219335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.219280 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-9c79h" podStartSLOduration=4.566256934 podStartE2EDuration="7.219262174s" totalImagesPullingTime="2.65300524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:46.218569112 +0000 UTC m=+76.944888069" watchObservedRunningTime="2026-09-10 16:18:46.219262174 +0000 UTC m=+76.945581125" Sep 10 16:18:46.246827 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.246774 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7748f44994-6bksq" podStartSLOduration=4.634498371 podStartE2EDuration="7.246757237s" totalImagesPullingTime="2.612258866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:46.244236185 +0000 UTC m=+76.970555116" watchObservedRunningTime="2026-09-10 16:18:46.246757237 +0000 UTC m=+76.973076158" Sep 10 16:18:46.267205 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.267142 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" podStartSLOduration=4.954446121 podStartE2EDuration="8.267123465s" totalImagesPullingTime="3.312677344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:46.266180983 +0000 UTC m=+76.992499914" watchObservedRunningTime="2026-09-10 16:18:46.267123465 +0000 UTC m=+76.993442395" Sep 10 16:18:46.294673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.294599 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" podStartSLOduration=2.148964112 podStartE2EDuration="8.294578287s" totalImagesPullingTime="6.145614175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:46.293467037 +0000 UTC m=+77.019785975" watchObservedRunningTime="2026-09-10 16:18:46.294578287 +0000 UTC m=+77.020897219" Sep 10 16:18:46.723891 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.723859 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:46.823827 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:46.823780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:47.764405 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:47.764371 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:18:48.194297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:48.194264 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerStarted","Data":"896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786"} Sep 10 16:18:48.194450 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:48.194333 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerStarted","Data":"a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c"} Sep 10 16:18:48.194450 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:48.194365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerStarted","Data":"5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1"} Sep 10 16:18:49.201944 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:49.201911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerStarted","Data":"4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9"} Sep 10 16:18:49.202345 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:49.201974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerStarted","Data":"7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67"} Sep 10 16:18:49.202345 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:49.201999 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerStarted","Data":"d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06"} Sep 10 16:18:49.237787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:49.237728 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.430213405 podStartE2EDuration="9.237707002s" totalImagesPullingTime="4.78249629s" totalInitContainerRuntime="24.997307ms" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:49.235446739 +0000 UTC m=+79.961765671" watchObservedRunningTime="2026-09-10 16:18:49.237707002 +0000 UTC m=+79.964025957" Sep 10 16:18:50.165240 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:50.165206 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:50.226363 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:50.226322 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:51.078818 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:51.078784 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:18:51.434017 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:51.433939 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:51.534305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:51.534277 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:51.635150 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:51.635113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:52.192298 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:52.192173 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-546cf76c59-9w54t" Sep 10 16:18:55.764813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:55.764779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:55.868376 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:55.868344 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:56.121244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:56.121161 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:56.184010 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:56.183978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:57.769750 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:57.769716 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-665655dc88-52ktb"] Sep 10 16:18:58.071597 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:58.071511 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:58.128858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:58.128820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:59.065332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:59.065296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:59.133478 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:59.133442 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:59.133634 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:59.133518 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:18:59.165724 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:59.165692 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:59.670324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:59.670285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:59.758199 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:18:59.758163 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:08.486459 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.486392 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-745985f847-r7tgz" podUID="1df1be45-a88c-4a16-af73-2c4680432842" containerName="console" containerID="cri-o://5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941" gracePeriod=15 Sep 10 16:19:08.727710 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.727685 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:19:08.794895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.794855 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-oauth-config\") pod \"1df1be45-a88c-4a16-af73-2c4680432842\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " Sep 10 16:19:08.794895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.794896 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-trusted-ca-bundle\") pod \"1df1be45-a88c-4a16-af73-2c4680432842\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " Sep 10 16:19:08.795173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.794914 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-console-config\") pod \"1df1be45-a88c-4a16-af73-2c4680432842\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " Sep 10 16:19:08.795173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.794969 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-oauth-serving-cert\") pod \"1df1be45-a88c-4a16-af73-2c4680432842\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " Sep 10 16:19:08.795173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.794995 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-serving-cert\") pod \"1df1be45-a88c-4a16-af73-2c4680432842\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " Sep 10 16:19:08.795173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.795013 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-service-ca\") pod \"1df1be45-a88c-4a16-af73-2c4680432842\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " Sep 10 16:19:08.795173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.795046 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8nks\" (UniqueName: \"kubernetes.io/projected/1df1be45-a88c-4a16-af73-2c4680432842-kube-api-access-l8nks\") pod \"1df1be45-a88c-4a16-af73-2c4680432842\" (UID: \"1df1be45-a88c-4a16-af73-2c4680432842\") " Sep 10 16:19:08.795428 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.795378 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1df1be45-a88c-4a16-af73-2c4680432842" (UID: "1df1be45-a88c-4a16-af73-2c4680432842"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:08.795483 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.795418 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-console-config" (OuterVolumeSpecName: "console-config") pod "1df1be45-a88c-4a16-af73-2c4680432842" (UID: "1df1be45-a88c-4a16-af73-2c4680432842"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:08.795483 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.795430 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1df1be45-a88c-4a16-af73-2c4680432842" (UID: "1df1be45-a88c-4a16-af73-2c4680432842"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:08.795483 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.795444 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-service-ca" (OuterVolumeSpecName: "service-ca") pod "1df1be45-a88c-4a16-af73-2c4680432842" (UID: "1df1be45-a88c-4a16-af73-2c4680432842"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:08.797382 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.797352 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1df1be45-a88c-4a16-af73-2c4680432842" (UID: "1df1be45-a88c-4a16-af73-2c4680432842"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:08.797382 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.797368 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df1be45-a88c-4a16-af73-2c4680432842-kube-api-access-l8nks" (OuterVolumeSpecName: "kube-api-access-l8nks") pod "1df1be45-a88c-4a16-af73-2c4680432842" (UID: "1df1be45-a88c-4a16-af73-2c4680432842"). InnerVolumeSpecName "kube-api-access-l8nks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:19:08.797514 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.797403 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1df1be45-a88c-4a16-af73-2c4680432842" (UID: "1df1be45-a88c-4a16-af73-2c4680432842"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:08.895793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.895758 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-console-config\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:08.895793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.895788 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-oauth-serving-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:08.895793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.895798 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-serving-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:08.896018 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.895808 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:08.896018 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.895816 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8nks\" (UniqueName: \"kubernetes.io/projected/1df1be45-a88c-4a16-af73-2c4680432842-kube-api-access-l8nks\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:08.896018 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.895824 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1df1be45-a88c-4a16-af73-2c4680432842-console-oauth-config\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:08.896018 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:08.895835 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df1be45-a88c-4a16-af73-2c4680432842-trusted-ca-bundle\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:09.295030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.294996 2562 generic.go:356] "Generic (PLEG): container finished" podID="1df1be45-a88c-4a16-af73-2c4680432842" containerID="5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941" exitCode=0 Sep 10 16:19:09.295244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.295062 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745985f847-r7tgz" Sep 10 16:19:09.295244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.295075 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745985f847-r7tgz" event={"ID":"1df1be45-a88c-4a16-af73-2c4680432842","Type":"ContainerDied","Data":"5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941"} Sep 10 16:19:09.295244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.295129 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745985f847-r7tgz" event={"ID":"1df1be45-a88c-4a16-af73-2c4680432842","Type":"ContainerDied","Data":"0237ff0a5e4ec5c2bcfa31b85ddac88ba07c2d6fa3d5176f3f04c5c00434beb7"} Sep 10 16:19:09.295244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.295160 2562 scope.go:117] "RemoveContainer" containerID="5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941" Sep 10 16:19:09.304306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.304284 2562 scope.go:117] "RemoveContainer" containerID="5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941" Sep 10 16:19:09.304613 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:09.304593 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941\": container with ID starting with 5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941 not found: ID does not exist" containerID="5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941" Sep 10 16:19:09.304674 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.304624 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941"} err="failed to get container status \"5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941\": rpc error: code = NotFound desc = could not find container \"5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941\": container with ID starting with 5f0612618f73a845d45e870ac7135a6fc625801110332d3e60f70a1129dcd941 not found: ID does not exist" Sep 10 16:19:09.317270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.317246 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-745985f847-r7tgz"] Sep 10 16:19:09.320668 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.320647 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-745985f847-r7tgz"] Sep 10 16:19:09.758575 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:09.758539 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1df1be45-a88c-4a16-af73-2c4680432842" path="/var/lib/kubelet/pods/1df1be45-a88c-4a16-af73-2c4680432842/volumes" Sep 10 16:19:14.154820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:14.154784 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mvhbl" Sep 10 16:19:14.667008 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:14.666973 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:14.768295 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:14.768260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:19.137820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:19.137788 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:19:19.141736 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:19.141712 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69f846fc4-5gghq" Sep 10 16:19:22.652510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:22.652473 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:22.753541 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:22.753506 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:22.795392 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:22.795343 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-665655dc88-52ktb" podUID="724b46cc-9941-422b-b2c8-05ed9414a031" containerName="console" containerID="cri-o://ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584" gracePeriod=15 Sep 10 16:19:23.025965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.025939 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:19:23.118722 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.118664 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-console-config\") pod \"724b46cc-9941-422b-b2c8-05ed9414a031\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " Sep 10 16:19:23.118722 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.118731 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-oauth-config\") pod \"724b46cc-9941-422b-b2c8-05ed9414a031\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " Sep 10 16:19:23.118980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.118798 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-serving-cert\") pod \"724b46cc-9941-422b-b2c8-05ed9414a031\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " Sep 10 16:19:23.118980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.118833 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd6zj\" (UniqueName: \"kubernetes.io/projected/724b46cc-9941-422b-b2c8-05ed9414a031-kube-api-access-vd6zj\") pod \"724b46cc-9941-422b-b2c8-05ed9414a031\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " Sep 10 16:19:23.118980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.118864 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-service-ca\") pod \"724b46cc-9941-422b-b2c8-05ed9414a031\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " Sep 10 16:19:23.118980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.118888 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-oauth-serving-cert\") pod \"724b46cc-9941-422b-b2c8-05ed9414a031\" (UID: \"724b46cc-9941-422b-b2c8-05ed9414a031\") " Sep 10 16:19:23.119305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.119270 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-console-config" (OuterVolumeSpecName: "console-config") pod "724b46cc-9941-422b-b2c8-05ed9414a031" (UID: "724b46cc-9941-422b-b2c8-05ed9414a031"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:23.119434 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.119399 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-service-ca" (OuterVolumeSpecName: "service-ca") pod "724b46cc-9941-422b-b2c8-05ed9414a031" (UID: "724b46cc-9941-422b-b2c8-05ed9414a031"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:23.119434 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.119422 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "724b46cc-9941-422b-b2c8-05ed9414a031" (UID: "724b46cc-9941-422b-b2c8-05ed9414a031"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:23.121208 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.121176 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "724b46cc-9941-422b-b2c8-05ed9414a031" (UID: "724b46cc-9941-422b-b2c8-05ed9414a031"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:23.121297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.121232 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724b46cc-9941-422b-b2c8-05ed9414a031-kube-api-access-vd6zj" (OuterVolumeSpecName: "kube-api-access-vd6zj") pod "724b46cc-9941-422b-b2c8-05ed9414a031" (UID: "724b46cc-9941-422b-b2c8-05ed9414a031"). InnerVolumeSpecName "kube-api-access-vd6zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:19:23.121297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.121248 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "724b46cc-9941-422b-b2c8-05ed9414a031" (UID: "724b46cc-9941-422b-b2c8-05ed9414a031"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:23.220320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.220225 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-serving-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:23.220320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.220261 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd6zj\" (UniqueName: \"kubernetes.io/projected/724b46cc-9941-422b-b2c8-05ed9414a031-kube-api-access-vd6zj\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:23.220320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.220273 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:23.220320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.220281 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-oauth-serving-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:23.220320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.220290 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724b46cc-9941-422b-b2c8-05ed9414a031-console-config\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:23.220320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.220298 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724b46cc-9941-422b-b2c8-05ed9414a031-console-oauth-config\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:23.357918 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.357887 2562 generic.go:356] "Generic (PLEG): container finished" podID="724b46cc-9941-422b-b2c8-05ed9414a031" containerID="ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584" exitCode=0 Sep 10 16:19:23.358096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.357931 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665655dc88-52ktb" Sep 10 16:19:23.358096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.357969 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665655dc88-52ktb" event={"ID":"724b46cc-9941-422b-b2c8-05ed9414a031","Type":"ContainerDied","Data":"ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584"} Sep 10 16:19:23.358096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.358019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665655dc88-52ktb" event={"ID":"724b46cc-9941-422b-b2c8-05ed9414a031","Type":"ContainerDied","Data":"fc2ca665c0d7d89b8826b3730a3510ee415ebed805e08d8ca90d1e453ca08d61"} Sep 10 16:19:23.358096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.358066 2562 scope.go:117] "RemoveContainer" containerID="ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584" Sep 10 16:19:23.366979 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.366960 2562 scope.go:117] "RemoveContainer" containerID="ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584" Sep 10 16:19:23.367290 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:23.367273 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584\": container with ID starting with ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584 not found: ID does not exist" containerID="ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584" Sep 10 16:19:23.367350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.367299 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584"} err="failed to get container status \"ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584\": rpc error: code = NotFound desc = could not find container \"ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584\": container with ID starting with ce20c8efb494a2501f05de6b21ae332fe148e72362bce31ae2db1cd561e2a584 not found: ID does not exist" Sep 10 16:19:23.380320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.380294 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-665655dc88-52ktb"] Sep 10 16:19:23.383512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.383490 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-665655dc88-52ktb"] Sep 10 16:19:23.756799 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.756761 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="724b46cc-9941-422b-b2c8-05ed9414a031" path="/var/lib/kubelet/pods/724b46cc-9941-422b-b2c8-05ed9414a031/volumes" Sep 10 16:19:23.955595 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:23.955561 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:24.009577 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:24.009499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:28.626323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:28.626291 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:28.694983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:28.694943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:41.078621 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:41.078578 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:19:41.098579 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:41.098549 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:19:41.451120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:41.451026 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:19:50.609281 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:50.609245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:50.670609 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:50.670573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:54.854056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:54.854007 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:19:54.854443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:54.854295 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="alertmanager" containerID="cri-o://d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716" gracePeriod=120 Sep 10 16:19:54.854443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:54.854361 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy-metric" containerID="cri-o://bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd" gracePeriod=120 Sep 10 16:19:54.854443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:54.854413 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="prom-label-proxy" containerID="cri-o://3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c" gracePeriod=120 Sep 10 16:19:54.854602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:54.854390 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="config-reloader" containerID="cri-o://7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390" gracePeriod=120 Sep 10 16:19:54.854602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:54.854371 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy-web" containerID="cri-o://464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5" gracePeriod=120 Sep 10 16:19:54.854602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:54.854420 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy" containerID="cri-o://410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56" gracePeriod=120 Sep 10 16:19:55.492770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492737 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerID="3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c" exitCode=0 Sep 10 16:19:55.492770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492765 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerID="bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd" exitCode=0 Sep 10 16:19:55.492770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492772 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerID="410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56" exitCode=0 Sep 10 16:19:55.492770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492779 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerID="7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390" exitCode=0 Sep 10 16:19:55.493135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492785 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerID="d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716" exitCode=0 Sep 10 16:19:55.493135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerDied","Data":"3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c"} Sep 10 16:19:55.493135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerDied","Data":"bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd"} Sep 10 16:19:55.493135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492895 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerDied","Data":"410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56"} Sep 10 16:19:55.493135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492919 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerDied","Data":"7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390"} Sep 10 16:19:55.493135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.492945 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerDied","Data":"d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716"} Sep 10 16:19:55.777187 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.777152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:55.920519 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.920474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:55.978262 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:55.978231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:56.103570 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.103547 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.117739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.117699 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-web-config\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.117739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.117747 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-volume\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.117958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.117812 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-out\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.117958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.117838 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.117958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.117865 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-main-db\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.117958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.117888 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-metrics-client-ca\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.117958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.117922 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpmw2\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-kube-api-access-rpmw2\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.117958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.117955 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.118286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.118006 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.118286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.118060 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-cluster-tls-config\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.118286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.118089 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.118286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.118145 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-tls-assets\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.118286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.118175 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9a5bf842-e92d-4b52-a664-6de26c861d6f\" (UID: \"9a5bf842-e92d-4b52-a664-6de26c861d6f\") " Sep 10 16:19:56.118892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.118810 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:56.119023 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.118904 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:19:56.119576 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.119546 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:19:56.122454 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.122412 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:56.122775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.122740 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-out" (OuterVolumeSpecName: "config-out") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:19:56.122873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.122825 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-kube-api-access-rpmw2" (OuterVolumeSpecName: "kube-api-access-rpmw2") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "kube-api-access-rpmw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:19:56.122873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.122827 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-volume" (OuterVolumeSpecName: "config-volume") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:56.122989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.122947 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:56.123092 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.123066 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:56.123310 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.123287 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:56.123370 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.123315 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:19:56.127334 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.127309 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:56.135149 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.134994 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-web-config" (OuterVolumeSpecName: "web-config") pod "9a5bf842-e92d-4b52-a664-6de26c861d6f" (UID: "9a5bf842-e92d-4b52-a664-6de26c861d6f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:19:56.219152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219112 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219141 2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-cluster-tls-config\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219153 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219162 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-tls-assets\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219171 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219183 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-web-config\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219191 2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219200 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-config-out\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219208 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-main-tls\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219215 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9a5bf842-e92d-4b52-a664-6de26c861d6f-alertmanager-main-db\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219224 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a5bf842-e92d-4b52-a664-6de26c861d6f-metrics-client-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219232 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmw2\" (UniqueName: \"kubernetes.io/projected/9a5bf842-e92d-4b52-a664-6de26c861d6f-kube-api-access-rpmw2\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.219409 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.219240 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a5bf842-e92d-4b52-a664-6de26c861d6f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:19:56.500557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.500516 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerID="464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5" exitCode=0 Sep 10 16:19:56.500714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.500571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerDied","Data":"464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5"} Sep 10 16:19:56.500714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.500589 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.500714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.500637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9a5bf842-e92d-4b52-a664-6de26c861d6f","Type":"ContainerDied","Data":"44f35306976d598353909b060fc350b482b474d79d80b1bbe267940ba228998c"} Sep 10 16:19:56.500714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.500674 2562 scope.go:117] "RemoveContainer" containerID="3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c" Sep 10 16:19:56.508716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.508575 2562 scope.go:117] "RemoveContainer" containerID="bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd" Sep 10 16:19:56.516200 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.516179 2562 scope.go:117] "RemoveContainer" containerID="410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56" Sep 10 16:19:56.522943 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.522923 2562 scope.go:117] "RemoveContainer" containerID="464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5" Sep 10 16:19:56.527736 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.527713 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:19:56.528554 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.528533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:56.531942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.531919 2562 scope.go:117] "RemoveContainer" containerID="7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390" Sep 10 16:19:56.533597 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.533574 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:19:56.539411 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.539394 2562 scope.go:117] "RemoveContainer" containerID="d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716" Sep 10 16:19:56.546374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.546354 2562 scope.go:117] "RemoveContainer" containerID="66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80" Sep 10 16:19:56.553332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.553315 2562 scope.go:117] "RemoveContainer" containerID="3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c" Sep 10 16:19:56.553605 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:56.553585 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c\": container with ID starting with 3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c not found: ID does not exist" containerID="3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c" Sep 10 16:19:56.553678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.553619 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c"} err="failed to get container status \"3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c\": rpc error: code = NotFound desc = could not find container \"3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c\": container with ID starting with 3da471bbf2cc15cd7b5a8cb3871fa032a227e8094de6a65dd3848b9753b8b38c not found: ID does not exist" Sep 10 16:19:56.553678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.553646 2562 scope.go:117] "RemoveContainer" containerID="bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd" Sep 10 16:19:56.553940 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:56.553914 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd\": container with ID starting with bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd not found: ID does not exist" containerID="bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd" Sep 10 16:19:56.553991 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.553939 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd"} err="failed to get container status \"bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd\": rpc error: code = NotFound desc = could not find container \"bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd\": container with ID starting with bdfa814bfd4e420ef6cce43425f3e3028fb5d1f74e259306add1d562c77743dd not found: ID does not exist" Sep 10 16:19:56.553991 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.553955 2562 scope.go:117] "RemoveContainer" containerID="410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56" Sep 10 16:19:56.554231 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:56.554215 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56\": container with ID starting with 410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56 not found: ID does not exist" containerID="410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56" Sep 10 16:19:56.554275 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.554238 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56"} err="failed to get container status \"410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56\": rpc error: code = NotFound desc = could not find container \"410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56\": container with ID starting with 410d5e5a49aa2fcc4e03160aef38cfd022ea02c1386e25a7dddd7ceb679f6c56 not found: ID does not exist" Sep 10 16:19:56.554275 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.554253 2562 scope.go:117] "RemoveContainer" containerID="464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5" Sep 10 16:19:56.554486 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:56.554471 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5\": container with ID starting with 464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5 not found: ID does not exist" containerID="464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5" Sep 10 16:19:56.554560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.554491 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5"} err="failed to get container status \"464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5\": rpc error: code = NotFound desc = could not find container \"464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5\": container with ID starting with 464cc6e08fb2003cccacf7985dc69a2d7c0e35972abe6a2ac9936e1df3ee81c5 not found: ID does not exist" Sep 10 16:19:56.554560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.554508 2562 scope.go:117] "RemoveContainer" containerID="7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390" Sep 10 16:19:56.554723 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:56.554709 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390\": container with ID starting with 7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390 not found: ID does not exist" containerID="7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390" Sep 10 16:19:56.554762 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.554726 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390"} err="failed to get container status \"7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390\": rpc error: code = NotFound desc = could not find container \"7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390\": container with ID starting with 7c0d3cbcb977f2d1e2043b53e1134fe7ee75a5c6ffcddc964202bb752b8b5390 not found: ID does not exist" Sep 10 16:19:56.554762 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.554738 2562 scope.go:117] "RemoveContainer" containerID="d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716" Sep 10 16:19:56.554976 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:56.554960 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716\": container with ID starting with d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716 not found: ID does not exist" containerID="d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716" Sep 10 16:19:56.555123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.554984 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716"} err="failed to get container status \"d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716\": rpc error: code = NotFound desc = could not find container \"d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716\": container with ID starting with d22e3cc73f8f6d953b334eb8b9b0f0264ffeda2042bcfaa81301a47856cf5716 not found: ID does not exist" Sep 10 16:19:56.555123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.555003 2562 scope.go:117] "RemoveContainer" containerID="66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80" Sep 10 16:19:56.555296 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:19:56.555281 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80\": container with ID starting with 66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80 not found: ID does not exist" containerID="66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80" Sep 10 16:19:56.555330 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.555302 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80"} err="failed to get container status \"66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80\": rpc error: code = NotFound desc = could not find container \"66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80\": container with ID starting with 66cece50c0d666a23a40e1f196d99c024a1ec8a75ac26fd31636acaa6e9ceb80 not found: ID does not exist" Sep 10 16:19:56.578261 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578229 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:19:56.578609 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578595 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df1be45-a88c-4a16-af73-2c4680432842" containerName="console" Sep 10 16:19:56.578652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578611 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df1be45-a88c-4a16-af73-2c4680432842" containerName="console" Sep 10 16:19:56.578652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578627 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724b46cc-9941-422b-b2c8-05ed9414a031" containerName="console" Sep 10 16:19:56.578652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578633 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="724b46cc-9941-422b-b2c8-05ed9414a031" containerName="console" Sep 10 16:19:56.578652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578639 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="alertmanager" Sep 10 16:19:56.578652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578647 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="alertmanager" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578659 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578666 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578680 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="init-config-reloader" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578687 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="init-config-reloader" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578693 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="config-reloader" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578698 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="config-reloader" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578707 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy-web" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578712 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy-web" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578718 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy-metric" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578723 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy-metric" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578732 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="prom-label-proxy" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578741 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="prom-label-proxy" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578785 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="config-reloader" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578794 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy-metric" Sep 10 16:19:56.578797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578799 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="alertmanager" Sep 10 16:19:56.579219 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578808 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy" Sep 10 16:19:56.579219 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578817 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="prom-label-proxy" Sep 10 16:19:56.579219 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578826 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="724b46cc-9941-422b-b2c8-05ed9414a031" containerName="console" Sep 10 16:19:56.579219 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578834 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" containerName="kube-rbac-proxy-web" Sep 10 16:19:56.579219 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.578839 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1df1be45-a88c-4a16-af73-2c4680432842" containerName="console" Sep 10 16:19:56.582699 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.582674 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.585827 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.585798 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 16:19:56.585952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.585800 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 16:19:56.585952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.585844 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 16:19:56.585952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.585859 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 16:19:56.585952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.585844 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 16:19:56.586204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.586000 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 16:19:56.586204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.586197 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 16:19:56.586300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.586200 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 16:19:56.586342 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.586314 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nsnhw\"" Sep 10 16:19:56.590731 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.590714 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 16:19:56.597987 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.597964 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:19:56.622735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622704 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-config-volume\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.622897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622752 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42f6a402-417c-46b3-b207-800ecf4e0405-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.622897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.622897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622845 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42f6a402-417c-46b3-b207-800ecf4e0405-config-out\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.622897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f6a402-417c-46b3-b207-800ecf4e0405-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.623027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.623027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.623027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f6a402-417c-46b3-b207-800ecf4e0405-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.623027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxw4\" (UniqueName: \"kubernetes.io/projected/42f6a402-417c-46b3-b207-800ecf4e0405-kube-api-access-hjxw4\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.623027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.622972 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.623194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.623057 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42f6a402-417c-46b3-b207-800ecf4e0405-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.623194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.623102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.623194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.623125 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-web-config\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.625234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.625211 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:56.723703 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.723658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42f6a402-417c-46b3-b207-800ecf4e0405-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.723882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.723723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.723882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.723758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-web-config\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.723882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.723788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-config-volume\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.723882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.723834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42f6a402-417c-46b3-b207-800ecf4e0405-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.723882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.723870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.724160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.723905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42f6a402-417c-46b3-b207-800ecf4e0405-config-out\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.724160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.723980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f6a402-417c-46b3-b207-800ecf4e0405-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.724160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.724028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.724160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.724084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.724160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.724114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f6a402-417c-46b3-b207-800ecf4e0405-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.724160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.724139 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxw4\" (UniqueName: \"kubernetes.io/projected/42f6a402-417c-46b3-b207-800ecf4e0405-kube-api-access-hjxw4\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.724477 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.724167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.724477 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.724428 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42f6a402-417c-46b3-b207-800ecf4e0405-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.726792 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.726758 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42f6a402-417c-46b3-b207-800ecf4e0405-config-out\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.727531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727241 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.727531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f6a402-417c-46b3-b207-800ecf4e0405-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.727531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727241 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.727531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f6a402-417c-46b3-b207-800ecf4e0405-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.727531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.727531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42f6a402-417c-46b3-b207-800ecf4e0405-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.727919 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727685 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-config-volume\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.727919 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.728057 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.727986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-web-config\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.728836 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.728815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42f6a402-417c-46b3-b207-800ecf4e0405-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.735810 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.735787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxw4\" (UniqueName: \"kubernetes.io/projected/42f6a402-417c-46b3-b207-800ecf4e0405-kube-api-access-hjxw4\") pod \"alertmanager-main-0\" (UID: \"42f6a402-417c-46b3-b207-800ecf4e0405\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:56.892671 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:56.892576 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 16:19:57.185391 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:57.185366 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 16:19:57.187508 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:19:57.187480 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f6a402_417c_46b3_b207_800ecf4e0405.slice/crio-2cbc160874771dc733ff8834b54572a8afcd327c78757f8b0d38c4777b250134 WatchSource:0}: Error finding container 2cbc160874771dc733ff8834b54572a8afcd327c78757f8b0d38c4777b250134: Status 404 returned error can't find the container with id 2cbc160874771dc733ff8834b54572a8afcd327c78757f8b0d38c4777b250134 Sep 10 16:19:57.505361 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:57.505326 2562 generic.go:356] "Generic (PLEG): container finished" podID="42f6a402-417c-46b3-b207-800ecf4e0405" containerID="ba8082ff377b576a11325d0a4c2d0d92768242ba86517d0b0a4bafd61ac06ab0" exitCode=0 Sep 10 16:19:57.505533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:57.505415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42f6a402-417c-46b3-b207-800ecf4e0405","Type":"ContainerDied","Data":"ba8082ff377b576a11325d0a4c2d0d92768242ba86517d0b0a4bafd61ac06ab0"} Sep 10 16:19:57.505533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:57.505501 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42f6a402-417c-46b3-b207-800ecf4e0405","Type":"ContainerStarted","Data":"2cbc160874771dc733ff8834b54572a8afcd327c78757f8b0d38c4777b250134"} Sep 10 16:19:57.756442 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:57.756369 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a5bf842-e92d-4b52-a664-6de26c861d6f" path="/var/lib/kubelet/pods/9a5bf842-e92d-4b52-a664-6de26c861d6f/volumes" Sep 10 16:19:58.514482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:58.514434 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42f6a402-417c-46b3-b207-800ecf4e0405","Type":"ContainerStarted","Data":"739a2521712f8e7701c7890de4cc51f195159675e496531d6f4b3f402c858cf0"} Sep 10 16:19:58.514830 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:58.514566 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42f6a402-417c-46b3-b207-800ecf4e0405","Type":"ContainerStarted","Data":"c161b2a0f53750289ab656befb67df5c9161a7154e5c128690dcd080c9f2293b"} Sep 10 16:19:58.514830 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:58.514608 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42f6a402-417c-46b3-b207-800ecf4e0405","Type":"ContainerStarted","Data":"f4221d782422ed5d134ffaf33e6cdc6480026682476432e4a7c37879361b58d3"} Sep 10 16:19:58.514830 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:58.514649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42f6a402-417c-46b3-b207-800ecf4e0405","Type":"ContainerStarted","Data":"1a50e1b4b38d578d5c1964deddb279e06b1e2f362c67c76014230f59e4b41a97"} Sep 10 16:19:58.514830 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:58.514684 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42f6a402-417c-46b3-b207-800ecf4e0405","Type":"ContainerStarted","Data":"fdb12eb7ef1051218105324841016cfa4baf4507216a703c4b781bda1de8c668"} Sep 10 16:19:58.514830 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:58.514718 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42f6a402-417c-46b3-b207-800ecf4e0405","Type":"ContainerStarted","Data":"02fc9f7975b72dafb3b392eb325016b8d2d09c2f2d3b9afbcd703ea40d067510"} Sep 10 16:19:58.546089 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:58.546018 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.531378958 podStartE2EDuration="2.54600163s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.622672ms" isStatefulPod=false podCreationTimestamp="2026-09-10 16:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:58.543417428 +0000 UTC m=+149.269736377" watchObservedRunningTime="2026-09-10 16:19:58.54600163 +0000 UTC m=+149.272320560" Sep 10 16:19:59.093880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.093844 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:19:59.094235 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.094179 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="prometheus" containerID="cri-o://5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1" gracePeriod=600 Sep 10 16:19:59.094235 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.094192 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy" containerID="cri-o://7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67" gracePeriod=600 Sep 10 16:19:59.094451 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.094241 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="config-reloader" containerID="cri-o://a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c" gracePeriod=600 Sep 10 16:19:59.094451 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.094209 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="thanos-sidecar" containerID="cri-o://896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786" gracePeriod=600 Sep 10 16:19:59.094451 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.094244 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy-web" containerID="cri-o://d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06" gracePeriod=600 Sep 10 16:19:59.094601 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.094202 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9" gracePeriod=600 Sep 10 16:19:59.522241 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522206 2562 generic.go:356] "Generic (PLEG): container finished" podID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerID="4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9" exitCode=0 Sep 10 16:19:59.522241 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522233 2562 generic.go:356] "Generic (PLEG): container finished" podID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerID="7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67" exitCode=0 Sep 10 16:19:59.522241 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522240 2562 generic.go:356] "Generic (PLEG): container finished" podID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerID="896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786" exitCode=0 Sep 10 16:19:59.522241 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522245 2562 generic.go:356] "Generic (PLEG): container finished" podID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerID="a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c" exitCode=0 Sep 10 16:19:59.522241 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522251 2562 generic.go:356] "Generic (PLEG): container finished" podID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerID="5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1" exitCode=0 Sep 10 16:19:59.522743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522273 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerDied","Data":"4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9"} Sep 10 16:19:59.522743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522338 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerDied","Data":"7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67"} Sep 10 16:19:59.522743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerDied","Data":"896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786"} Sep 10 16:19:59.522743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522389 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerDied","Data":"a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c"} Sep 10 16:19:59.522743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:19:59.522412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerDied","Data":"5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1"} Sep 10 16:20:00.340307 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.340280 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.358537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358510 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-trusted-ca-bundle\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.358726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358551 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-grpc-tls\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.358726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358567 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-tls-assets\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.358726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358586 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-config-out\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.358726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358604 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.358726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358663 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-config\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.358726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358705 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358735 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-2\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358769 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-web-config\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358797 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-1\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-metrics-client-ca\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358877 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-kube-rbac-proxy\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358904 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-thanos-prometheus-http-client-file\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.358991 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbr6b\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-kube-api-access-wbr6b\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359006 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:00.359435 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359051 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-db\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.359435 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359325 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359676 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359830 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359883 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-serving-certs-ca-bundle\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359921 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-0\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359948 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-kubelet-serving-ca-bundle\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.359976 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-metrics-client-certs\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.360010 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-tls\") pod \"a87c0e89-0ac4-4d97-96f4-030e30776c36\" (UID: \"a87c0e89-0ac4-4d97-96f4-030e30776c36\") " Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.360442 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.360469 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.360484 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.360562 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-metrics-client-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.360639 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.360664 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:00.360887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.360740 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:00.362227 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.362188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-config-out" (OuterVolumeSpecName: "config-out") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:20:00.362349 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.362248 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:20:00.362718 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.362679 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.362871 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.362839 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-config" (OuterVolumeSpecName: "config") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.363267 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.363176 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.363346 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.363311 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:20:00.363461 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.363435 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.363758 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.363738 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.364316 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.364288 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.364393 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.364354 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.364708 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.364684 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-kube-api-access-wbr6b" (OuterVolumeSpecName: "kube-api-access-wbr6b") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "kube-api-access-wbr6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:20:00.364793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.364764 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.375128 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.375102 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-web-config" (OuterVolumeSpecName: "web-config") pod "a87c0e89-0ac4-4d97-96f4-030e30776c36" (UID: "a87c0e89-0ac4-4d97-96f4-030e30776c36"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:20:00.461207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461113 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-web-config\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461145 2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-kube-rbac-proxy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461161 2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461171 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbr6b\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-kube-api-access-wbr6b\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461183 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-db\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461192 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461201 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461212 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a87c0e89-0ac4-4d97-96f4-030e30776c36-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461220 2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-metrics-client-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461235 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461248 2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-grpc-tls\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461256 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a87c0e89-0ac4-4d97-96f4-030e30776c36-tls-assets\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461263 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a87c0e89-0ac4-4d97-96f4-030e30776c36-config-out\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461271 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461279 2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-config\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.461543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.461287 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a87c0e89-0ac4-4d97-96f4-030e30776c36-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:20:00.530324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.530288 2562 generic.go:356] "Generic (PLEG): container finished" podID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerID="d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06" exitCode=0 Sep 10 16:20:00.530780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.530363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerDied","Data":"d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06"} Sep 10 16:20:00.530780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.530389 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.530780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.530455 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a87c0e89-0ac4-4d97-96f4-030e30776c36","Type":"ContainerDied","Data":"e3f2950c18e2131458736ff7b0b77ad8483102938bb6c3cd200610b57f69362f"} Sep 10 16:20:00.530780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.530488 2562 scope.go:117] "RemoveContainer" containerID="4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9" Sep 10 16:20:00.539325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.539234 2562 scope.go:117] "RemoveContainer" containerID="7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67" Sep 10 16:20:00.546720 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.546704 2562 scope.go:117] "RemoveContainer" containerID="d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06" Sep 10 16:20:00.553813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.553793 2562 scope.go:117] "RemoveContainer" containerID="896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786" Sep 10 16:20:00.560745 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.560722 2562 scope.go:117] "RemoveContainer" containerID="a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c" Sep 10 16:20:00.568004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.567980 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:20:00.568375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.568358 2562 scope.go:117] "RemoveContainer" containerID="5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1" Sep 10 16:20:00.580474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.576283 2562 scope.go:117] "RemoveContainer" containerID="2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4" Sep 10 16:20:00.580474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.580380 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:20:00.587854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.587833 2562 scope.go:117] "RemoveContainer" containerID="4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9" Sep 10 16:20:00.588147 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:20:00.588126 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9\": container with ID starting with 4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9 not found: ID does not exist" containerID="4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9" Sep 10 16:20:00.588212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.588159 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9"} err="failed to get container status \"4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9\": rpc error: code = NotFound desc = could not find container \"4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9\": container with ID starting with 4ecf3dd06993d910aa789099c4a954b36c443e894b1236cb5daf5d754d163ed9 not found: ID does not exist" Sep 10 16:20:00.588212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.588178 2562 scope.go:117] "RemoveContainer" containerID="7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67" Sep 10 16:20:00.588458 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:20:00.588437 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67\": container with ID starting with 7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67 not found: ID does not exist" containerID="7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67" Sep 10 16:20:00.588495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.588465 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67"} err="failed to get container status \"7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67\": rpc error: code = NotFound desc = could not find container \"7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67\": container with ID starting with 7841b6b61ac3e744e811b9c7b08a6f65c28e58ba37892c1743b72d7cdda70a67 not found: ID does not exist" Sep 10 16:20:00.588495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.588482 2562 scope.go:117] "RemoveContainer" containerID="d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06" Sep 10 16:20:00.588741 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:20:00.588725 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06\": container with ID starting with d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06 not found: ID does not exist" containerID="d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06" Sep 10 16:20:00.588790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.588747 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06"} err="failed to get container status \"d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06\": rpc error: code = NotFound desc = could not find container \"d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06\": container with ID starting with d5fd511f3a7371f6782e8075ffc059df7b7267a02ec65198ca7680ffae24cb06 not found: ID does not exist" Sep 10 16:20:00.588790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.588760 2562 scope.go:117] "RemoveContainer" containerID="896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786" Sep 10 16:20:00.588983 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:20:00.588966 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786\": container with ID starting with 896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786 not found: ID does not exist" containerID="896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786" Sep 10 16:20:00.589072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.589000 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786"} err="failed to get container status \"896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786\": rpc error: code = NotFound desc = could not find container \"896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786\": container with ID starting with 896919f5147174b5da2788c70cd253fb8a51e18c7dd791686bdcdd7726716786 not found: ID does not exist" Sep 10 16:20:00.589072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.589020 2562 scope.go:117] "RemoveContainer" containerID="a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c" Sep 10 16:20:00.589313 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:20:00.589293 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c\": container with ID starting with a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c not found: ID does not exist" containerID="a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c" Sep 10 16:20:00.589396 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.589318 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c"} err="failed to get container status \"a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c\": rpc error: code = NotFound desc = could not find container \"a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c\": container with ID starting with a3e41a017ed54781f0970100afa051840d140a9690f4527bb2efa4e6b36df44c not found: ID does not exist" Sep 10 16:20:00.589396 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.589331 2562 scope.go:117] "RemoveContainer" containerID="5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1" Sep 10 16:20:00.589594 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:20:00.589575 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1\": container with ID starting with 5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1 not found: ID does not exist" containerID="5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1" Sep 10 16:20:00.589635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.589597 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1"} err="failed to get container status \"5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1\": rpc error: code = NotFound desc = could not find container \"5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1\": container with ID starting with 5c1515c4f60223df74717baa451d0aa63d2f0b2038bfe2ae6a165d0890bb57e1 not found: ID does not exist" Sep 10 16:20:00.589635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.589610 2562 scope.go:117] "RemoveContainer" containerID="2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4" Sep 10 16:20:00.589819 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:20:00.589803 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4\": container with ID starting with 2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4 not found: ID does not exist" containerID="2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4" Sep 10 16:20:00.589861 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.589823 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4"} err="failed to get container status \"2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4\": rpc error: code = NotFound desc = could not find container \"2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4\": container with ID starting with 2287ccb1a5f1eb9ef8f2580be464fb07dcb8ab2651654863b0ab69463a87f2b4 not found: ID does not exist" Sep 10 16:20:00.608900 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.608867 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:20:00.609216 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609202 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy-thanos" Sep 10 16:20:00.609263 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609218 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy-thanos" Sep 10 16:20:00.609263 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609229 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="prometheus" Sep 10 16:20:00.609263 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609235 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="prometheus" Sep 10 16:20:00.609263 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609255 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="thanos-sidecar" Sep 10 16:20:00.609263 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609261 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="thanos-sidecar" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609271 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="init-config-reloader" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609276 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="init-config-reloader" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609284 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="config-reloader" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609288 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="config-reloader" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609293 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609298 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609309 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy-web" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609314 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy-web" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609367 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="prometheus" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609375 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="config-reloader" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609380 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="thanos-sidecar" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609386 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy-thanos" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609393 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy" Sep 10 16:20:00.609402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.609402 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" containerName="kube-rbac-proxy-web" Sep 10 16:20:00.613764 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.613742 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.617712 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.617688 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 16:20:00.617888 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.617743 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 16:20:00.617888 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.617750 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 16:20:00.617888 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.617844 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bzbn4\"" Sep 10 16:20:00.618054 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.617887 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 16:20:00.618156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618124 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 16:20:00.618285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618188 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 16:20:00.618285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618197 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 16:20:00.618285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618211 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8t5desa4u6c0k\"" Sep 10 16:20:00.618551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618530 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 16:20:00.618645 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618555 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 16:20:00.618914 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618891 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 16:20:00.619008 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618946 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 16:20:00.619008 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.618987 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 16:20:00.620876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.620856 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 16:20:00.623474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.623459 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 16:20:00.630118 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.630095 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:20:00.662467 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662430 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efcd3fac-a22a-459f-8bb1-04f6165757c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbj9\" (UniqueName: \"kubernetes.io/projected/efcd3fac-a22a-459f-8bb1-04f6165757c3-kube-api-access-jcbj9\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efcd3fac-a22a-459f-8bb1-04f6165757c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-config\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662955 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.662985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.662988 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764247 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efcd3fac-a22a-459f-8bb1-04f6165757c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-config\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764806 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764806 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764806 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764789 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.764936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.765160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.764975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.765160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.765010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.765160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.765086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efcd3fac-a22a-459f-8bb1-04f6165757c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.765160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.765122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.765160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.765145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbj9\" (UniqueName: \"kubernetes.io/projected/efcd3fac-a22a-459f-8bb1-04f6165757c3-kube-api-access-jcbj9\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.765405 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.765191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.765405 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.765303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.766695 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.765690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.766695 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.766383 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.767386 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.767359 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efcd3fac-a22a-459f-8bb1-04f6165757c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.767584 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.767550 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.767691 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.767653 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.767812 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.767794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.767885 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.765086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.767937 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.767879 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.768292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.768259 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.768449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.768406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.768797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.768776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.768895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.768870 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.769365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.769344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-config\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.769912 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.769894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.769978 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.769914 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efcd3fac-a22a-459f-8bb1-04f6165757c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.770083 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.770064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efcd3fac-a22a-459f-8bb1-04f6165757c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.770842 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.770824 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efcd3fac-a22a-459f-8bb1-04f6165757c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.773550 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.773532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbj9\" (UniqueName: \"kubernetes.io/projected/efcd3fac-a22a-459f-8bb1-04f6165757c3-kube-api-access-jcbj9\") pod \"prometheus-k8s-0\" (UID: \"efcd3fac-a22a-459f-8bb1-04f6165757c3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:00.925918 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:00.925878 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:01.208840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:01.208809 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 16:20:01.209443 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:20:01.209408 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcd3fac_a22a_459f_8bb1_04f6165757c3.slice/crio-5dfe9e9c9fcdb61c499ee2017785c0ff73d8827b049719e0934ab2f26851e3b7 WatchSource:0}: Error finding container 5dfe9e9c9fcdb61c499ee2017785c0ff73d8827b049719e0934ab2f26851e3b7: Status 404 returned error can't find the container with id 5dfe9e9c9fcdb61c499ee2017785c0ff73d8827b049719e0934ab2f26851e3b7 Sep 10 16:20:01.537179 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:01.537142 2562 generic.go:356] "Generic (PLEG): container finished" podID="efcd3fac-a22a-459f-8bb1-04f6165757c3" containerID="0c2d47f5c975f723c3ee19f0e16277946bd5c63dfac758c0aa6637b9a91ad746" exitCode=0 Sep 10 16:20:01.537537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:01.537207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efcd3fac-a22a-459f-8bb1-04f6165757c3","Type":"ContainerDied","Data":"0c2d47f5c975f723c3ee19f0e16277946bd5c63dfac758c0aa6637b9a91ad746"} Sep 10 16:20:01.537537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:01.537275 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efcd3fac-a22a-459f-8bb1-04f6165757c3","Type":"ContainerStarted","Data":"5dfe9e9c9fcdb61c499ee2017785c0ff73d8827b049719e0934ab2f26851e3b7"} Sep 10 16:20:01.760803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:01.760021 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87c0e89-0ac4-4d97-96f4-030e30776c36" path="/var/lib/kubelet/pods/a87c0e89-0ac4-4d97-96f4-030e30776c36/volumes" Sep 10 16:20:02.545410 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.545370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efcd3fac-a22a-459f-8bb1-04f6165757c3","Type":"ContainerStarted","Data":"c65fdfe2d24db3dcab1a901823e066788caf63585405e100da7d711f13512927"} Sep 10 16:20:02.545853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.545443 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efcd3fac-a22a-459f-8bb1-04f6165757c3","Type":"ContainerStarted","Data":"de4c545d8a0eb5c2b633de087db97af400cb89b638a207020ea1956b88fd6259"} Sep 10 16:20:02.545853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.545471 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efcd3fac-a22a-459f-8bb1-04f6165757c3","Type":"ContainerStarted","Data":"d66d9f6f62a248f52b750b94b6f7894caca715947cf970a518f073f5db7ca415"} Sep 10 16:20:02.545853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.545494 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efcd3fac-a22a-459f-8bb1-04f6165757c3","Type":"ContainerStarted","Data":"dc27700aa8a923b4e3ea87d85fcf4b935284a0e6bad17da72387bc30516df291"} Sep 10 16:20:02.545853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.545518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efcd3fac-a22a-459f-8bb1-04f6165757c3","Type":"ContainerStarted","Data":"9c4f1b1100602f6a5337cee6e225298d8a9b69ebdf11257dc5b063324dc4d5df"} Sep 10 16:20:02.545853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.545542 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efcd3fac-a22a-459f-8bb1-04f6165757c3","Type":"ContainerStarted","Data":"a30c2e3a5123381c222999683083827d79afe613594c5977a83b7afaaa0a4d38"} Sep 10 16:20:02.581267 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.581207 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.561244885 podStartE2EDuration="2.581187611s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.942726ms" isStatefulPod=false podCreationTimestamp="2026-09-10 16:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:20:02.579409392 +0000 UTC m=+153.305728335" watchObservedRunningTime="2026-09-10 16:20:02.581187611 +0000 UTC m=+153.307506542" Sep 10 16:20:02.650339 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.650302 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:02.718997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:02.718955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:05.926602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:05.926556 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:20:07.196607 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:07.196569 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:07.296880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:07.296852 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:07.359246 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:07.359213 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:14.709808 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:14.709771 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:14.952811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:14.952773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:53.738419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:53.738347 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:58.836928 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:58.836896 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:20:58.905733 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:20:58.905699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:00.927088 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:00.927022 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:21:00.943186 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:00.943157 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:21:01.802963 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:01.802936 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 16:21:06.861279 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:06.861246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:06.951181 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:06.951152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:32.748790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:32.748752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:33.110477 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:33.110397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:36.780898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:36.780863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:36.882371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:36.882333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:38.849954 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:38.849914 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:38.990841 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:38.990809 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:39.133256 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:39.133167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:39.265350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:39.265312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:41.113850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.113816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:41.702858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.702820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:41.736292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.736258 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-klwmd"] Sep 10 16:21:41.738791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.738763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" Sep 10 16:21:41.742819 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.742796 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 16:21:41.744204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.744178 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kw9q8\"" Sep 10 16:21:41.744321 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.744221 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 16:21:41.757255 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.757229 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-klwmd"] Sep 10 16:21:41.853295 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.853258 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83446523-843f-423a-813b-3def176b07a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-klwmd\" (UID: \"83446523-843f-423a-813b-3def176b07a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" Sep 10 16:21:41.853459 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.853308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbs4x\" (UniqueName: \"kubernetes.io/projected/83446523-843f-423a-813b-3def176b07a4-kube-api-access-kbs4x\") pod \"cert-manager-cainjector-559cdd8c87-klwmd\" (UID: \"83446523-843f-423a-813b-3def176b07a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" Sep 10 16:21:41.954062 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.953965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83446523-843f-423a-813b-3def176b07a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-klwmd\" (UID: \"83446523-843f-423a-813b-3def176b07a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" Sep 10 16:21:41.954062 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.954029 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbs4x\" (UniqueName: \"kubernetes.io/projected/83446523-843f-423a-813b-3def176b07a4-kube-api-access-kbs4x\") pod \"cert-manager-cainjector-559cdd8c87-klwmd\" (UID: \"83446523-843f-423a-813b-3def176b07a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" Sep 10 16:21:41.964914 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.964887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83446523-843f-423a-813b-3def176b07a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-klwmd\" (UID: \"83446523-843f-423a-813b-3def176b07a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" Sep 10 16:21:41.965070 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:41.965030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbs4x\" (UniqueName: \"kubernetes.io/projected/83446523-843f-423a-813b-3def176b07a4-kube-api-access-kbs4x\") pod \"cert-manager-cainjector-559cdd8c87-klwmd\" (UID: \"83446523-843f-423a-813b-3def176b07a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" Sep 10 16:21:42.059292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:42.059239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" Sep 10 16:21:42.332538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:42.332512 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-klwmd"] Sep 10 16:21:42.334807 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:21:42.334774 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83446523_843f_423a_813b_3def176b07a4.slice/crio-0aedf3d56132a77c2213a145f0ef2ec88af88dee000a10bd38bab45fb5283cbe WatchSource:0}: Error finding container 0aedf3d56132a77c2213a145f0ef2ec88af88dee000a10bd38bab45fb5283cbe: Status 404 returned error can't find the container with id 0aedf3d56132a77c2213a145f0ef2ec88af88dee000a10bd38bab45fb5283cbe Sep 10 16:21:42.950850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:42.950810 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" event={"ID":"83446523-843f-423a-813b-3def176b07a4","Type":"ContainerStarted","Data":"0aedf3d56132a77c2213a145f0ef2ec88af88dee000a10bd38bab45fb5283cbe"} Sep 10 16:21:43.157137 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:43.157097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:43.232165 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:43.231999 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:45.620430 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:45.620389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:45.731900 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:45.731871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:45.966378 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:45.966290 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" event={"ID":"83446523-843f-423a-813b-3def176b07a4","Type":"ContainerStarted","Data":"50420bfaf475da8df28e1b0ecce607a8433b70bee8e40c27db22caf776161925"} Sep 10 16:21:45.986793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:45.986741 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-klwmd" podStartSLOduration=2.089704545 podStartE2EDuration="4.986725656s" totalImagesPullingTime="2.897021111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:21:45.985053754 +0000 UTC m=+256.711372684" watchObservedRunningTime="2026-09-10 16:21:45.986725656 +0000 UTC m=+256.713044585" Sep 10 16:21:50.127202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:50.127154 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:53.373320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:53.373221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:53.470710 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:53.470681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:55.000414 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.000382 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg"] Sep 10 16:21:55.003682 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.003662 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" Sep 10 16:21:55.006833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.006785 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 16:21:55.008214 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.008196 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-klqhs\"" Sep 10 16:21:55.008291 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.008244 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 16:21:55.014698 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.014675 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg"] Sep 10 16:21:55.170629 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.170588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx69j\" (UniqueName: \"kubernetes.io/projected/decffee1-2d2a-4fc9-9c45-9c7aaa184066-kube-api-access-gx69j\") pod \"openshift-lws-operator-fd8ccff4c-kchdg\" (UID: \"decffee1-2d2a-4fc9-9c45-9c7aaa184066\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" Sep 10 16:21:55.170629 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.170634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/decffee1-2d2a-4fc9-9c45-9c7aaa184066-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kchdg\" (UID: \"decffee1-2d2a-4fc9-9c45-9c7aaa184066\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" Sep 10 16:21:55.272206 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.272133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx69j\" (UniqueName: \"kubernetes.io/projected/decffee1-2d2a-4fc9-9c45-9c7aaa184066-kube-api-access-gx69j\") pod \"openshift-lws-operator-fd8ccff4c-kchdg\" (UID: \"decffee1-2d2a-4fc9-9c45-9c7aaa184066\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" Sep 10 16:21:55.272206 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.272180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/decffee1-2d2a-4fc9-9c45-9c7aaa184066-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kchdg\" (UID: \"decffee1-2d2a-4fc9-9c45-9c7aaa184066\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" Sep 10 16:21:55.272526 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.272510 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/decffee1-2d2a-4fc9-9c45-9c7aaa184066-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kchdg\" (UID: \"decffee1-2d2a-4fc9-9c45-9c7aaa184066\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" Sep 10 16:21:55.281947 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.281922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx69j\" (UniqueName: \"kubernetes.io/projected/decffee1-2d2a-4fc9-9c45-9c7aaa184066-kube-api-access-gx69j\") pod \"openshift-lws-operator-fd8ccff4c-kchdg\" (UID: \"decffee1-2d2a-4fc9-9c45-9c7aaa184066\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" Sep 10 16:21:55.312835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.312800 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" Sep 10 16:21:55.583921 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:55.583815 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg"] Sep 10 16:21:55.586285 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:21:55.586250 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddecffee1_2d2a_4fc9_9c45_9c7aaa184066.slice/crio-1c666cdf2c3cad11bc44c0a8585a9370a887dd93c9d8af5a4c274bbb698ae9d9 WatchSource:0}: Error finding container 1c666cdf2c3cad11bc44c0a8585a9370a887dd93c9d8af5a4c274bbb698ae9d9: Status 404 returned error can't find the container with id 1c666cdf2c3cad11bc44c0a8585a9370a887dd93c9d8af5a4c274bbb698ae9d9 Sep 10 16:21:56.006885 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:56.006849 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" event={"ID":"decffee1-2d2a-4fc9-9c45-9c7aaa184066","Type":"ContainerStarted","Data":"1c666cdf2c3cad11bc44c0a8585a9370a887dd93c9d8af5a4c274bbb698ae9d9"} Sep 10 16:21:59.023602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:59.023567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" event={"ID":"decffee1-2d2a-4fc9-9c45-9c7aaa184066","Type":"ContainerStarted","Data":"6656dfd550ff89192e1cf13640e5d02bed32317099a4beef8aaf740ee792b9f6"} Sep 10 16:21:59.044935 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:59.044883 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kchdg" podStartSLOduration=2.586398235 podStartE2EDuration="5.044865789s" totalImagesPullingTime="2.458467554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:21:59.043228701 +0000 UTC m=+269.769547633" watchObservedRunningTime="2026-09-10 16:21:59.044865789 +0000 UTC m=+269.771184719" Sep 10 16:21:59.848272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:59.848231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:21:59.948071 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:21:59.948020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:02.101323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:02.101288 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:03.579495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:03.579459 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:03.728781 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:03.728750 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:08.787892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:08.787857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:13.517806 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:13.517772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:14.806468 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.806435 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:14.820626 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.820595 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js"] Sep 10 16:22:14.894992 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.894949 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js"] Sep 10 16:22:14.895180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.895109 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:14.898700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.898679 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 16:22:14.898849 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.898725 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tbzcd\"" Sep 10 16:22:14.899180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.899165 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 16:22:14.899247 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.899206 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 16:22:14.899289 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.899263 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 16:22:14.936635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.936589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05e1d2a0-ef40-4578-ad49-639e172a3b11-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:14.936635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.936629 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmwmk\" (UniqueName: \"kubernetes.io/projected/05e1d2a0-ef40-4578-ad49-639e172a3b11-kube-api-access-hmwmk\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:14.936897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.936735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05e1d2a0-ef40-4578-ad49-639e172a3b11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:14.936897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.936771 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05e1d2a0-ef40-4578-ad49-639e172a3b11-tmp\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:14.936897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.936838 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05e1d2a0-ef40-4578-ad49-639e172a3b11-manifests\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:14.970998 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:14.970968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:15.037632 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.037590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05e1d2a0-ef40-4578-ad49-639e172a3b11-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.037632 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.037629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwmk\" (UniqueName: \"kubernetes.io/projected/05e1d2a0-ef40-4578-ad49-639e172a3b11-kube-api-access-hmwmk\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.037837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.037682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05e1d2a0-ef40-4578-ad49-639e172a3b11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.037837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.037702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05e1d2a0-ef40-4578-ad49-639e172a3b11-tmp\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.037837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.037734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05e1d2a0-ef40-4578-ad49-639e172a3b11-manifests\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.038232 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.038205 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05e1d2a0-ef40-4578-ad49-639e172a3b11-manifests\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.040129 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.040100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05e1d2a0-ef40-4578-ad49-639e172a3b11-tmp\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.040453 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.040431 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05e1d2a0-ef40-4578-ad49-639e172a3b11-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.040523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.040431 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05e1d2a0-ef40-4578-ad49-639e172a3b11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.048417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.048390 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwmk\" (UniqueName: \"kubernetes.io/projected/05e1d2a0-ef40-4578-ad49-639e172a3b11-kube-api-access-hmwmk\") pod \"opendatahub-operator-controller-manager-57b77968d9-922js\" (UID: \"05e1d2a0-ef40-4578-ad49-639e172a3b11\") " pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.074475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.074403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:15.178671 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.178631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:15.205679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.205643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:15.274456 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.274398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:15.393677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.393591 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:15.514672 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.514637 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js"] Sep 10 16:22:15.517900 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:22:15.517874 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e1d2a0_ef40_4578_ad49_639e172a3b11.slice/crio-a62b057283d796cc8d6a8c2bfd355d294d04d57f1bfcb8ecf64326719c75bca6 WatchSource:0}: Error finding container a62b057283d796cc8d6a8c2bfd355d294d04d57f1bfcb8ecf64326719c75bca6: Status 404 returned error can't find the container with id a62b057283d796cc8d6a8c2bfd355d294d04d57f1bfcb8ecf64326719c75bca6 Sep 10 16:22:15.530472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.530443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:15.647653 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:15.647565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:16.097166 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:16.097124 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" event={"ID":"05e1d2a0-ef40-4578-ad49-639e172a3b11","Type":"ContainerStarted","Data":"a62b057283d796cc8d6a8c2bfd355d294d04d57f1bfcb8ecf64326719c75bca6"} Sep 10 16:22:19.112878 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:19.112844 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" event={"ID":"05e1d2a0-ef40-4578-ad49-639e172a3b11","Type":"ContainerStarted","Data":"e6d2753f93965395f85e815b4ce3d2229f0e39265ab4aeb4b5e42f5ff844a17d"} Sep 10 16:22:20.119421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:20.119381 2562 generic.go:356] "Generic (PLEG): container finished" podID="05e1d2a0-ef40-4578-ad49-639e172a3b11" containerID="e6d2753f93965395f85e815b4ce3d2229f0e39265ab4aeb4b5e42f5ff844a17d" exitCode=0 Sep 10 16:22:20.119928 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:20.119437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" event={"ID":"05e1d2a0-ef40-4578-ad49-639e172a3b11","Type":"ContainerDied","Data":"e6d2753f93965395f85e815b4ce3d2229f0e39265ab4aeb4b5e42f5ff844a17d"} Sep 10 16:22:21.126176 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:21.126140 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" event={"ID":"05e1d2a0-ef40-4578-ad49-639e172a3b11","Type":"ContainerStarted","Data":"78031ca90fad8c6e899fcbfa4c4ad3d24eef30c57fde889cf415454746181047"} Sep 10 16:22:21.126561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:21.126274 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:21.161387 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:21.161338 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" podStartSLOduration=2.846081173 podStartE2EDuration="7.161320551s" totalImagesPullingTime="3.278113639s" totalInitContainerRuntime="1.037125739s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:21.160571374 +0000 UTC m=+291.886890305" watchObservedRunningTime="2026-09-10 16:22:21.161320551 +0000 UTC m=+291.887639545" Sep 10 16:22:24.245103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:24.245066 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:24.346623 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:24.346591 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:29.672325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:29.672289 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:22:29.675786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:29.675764 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:22:29.680899 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:29.680874 2562 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 16:22:32.133524 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:32.133487 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57b77968d9-922js" Sep 10 16:22:34.513767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:34.513726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:34.654403 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:34.654367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:40.131621 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:40.131587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:40.993895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:40.993848 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:41.605859 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:41.605820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:41.688110 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:41.688065 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:42.797462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:42.797391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:45.514403 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:45.514373 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:45.616124 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:45.616092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:45.717571 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:45.717532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:45.815439 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:45.815349 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:46.143839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:46.143748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:46.510242 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:46.510208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:47.765152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:47.765109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:47.882808 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:47.882767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:48.493530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.493499 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz"] Sep 10 16:22:48.498570 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.498542 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.501622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.501601 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 16:22:48.502290 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.502271 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b27k5\"" Sep 10 16:22:48.502399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.502379 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 16:22:48.502968 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.502951 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 16:22:48.514174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.514147 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz"] Sep 10 16:22:48.533541 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.533512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:48.635180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.635375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.635375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635221 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.635375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.635375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bxt\" (UniqueName: \"kubernetes.io/projected/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-kube-api-access-p5bxt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.635604 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635439 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.635604 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.635604 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635504 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.635604 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.635554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736631 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736631 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bxt\" (UniqueName: \"kubernetes.io/projected/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-kube-api-access-p5bxt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736764 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.736897 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.736806 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.737272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.737150 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.737272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.737187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.737272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.737239 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.737272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.737267 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.737671 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.737645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.739103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.739071 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.739430 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.739410 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.746768 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.746743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bxt\" (UniqueName: \"kubernetes.io/projected/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-kube-api-access-p5bxt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.747166 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.747124 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152ab912-d24c-4e5e-a0a6-ba768c0cb7ec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz\" (UID: \"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.809114 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.809074 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:48.832152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.832106 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:48.933272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:48.933239 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:49.114557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:49.114470 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz"] Sep 10 16:22:49.117406 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:22:49.117379 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152ab912_d24c_4e5e_a0a6_ba768c0cb7ec.slice/crio-743d7db30b8339fe46569b9839b6a68f0d05fcf0add23d9c192e4a0d3fd4bd4f WatchSource:0}: Error finding container 743d7db30b8339fe46569b9839b6a68f0d05fcf0add23d9c192e4a0d3fd4bd4f: Status 404 returned error can't find the container with id 743d7db30b8339fe46569b9839b6a68f0d05fcf0add23d9c192e4a0d3fd4bd4f Sep 10 16:22:49.122026 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:49.122002 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:22:49.254350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:49.254319 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" event={"ID":"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec","Type":"ContainerStarted","Data":"743d7db30b8339fe46569b9839b6a68f0d05fcf0add23d9c192e4a0d3fd4bd4f"} Sep 10 16:22:52.101332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:52.101290 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:22:52.101589 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:52.101376 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:22:52.101589 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:52.101411 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:22:52.271419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:52.271384 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" event={"ID":"152ab912-d24c-4e5e-a0a6-ba768c0cb7ec","Type":"ContainerStarted","Data":"b76172704ee5f7a968cc914878d740e4971e4f76ac4d34e107b8568b52470a24"} Sep 10 16:22:52.309174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:52.309119 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" podStartSLOduration=1.330269077 podStartE2EDuration="4.309101321s" totalImagesPullingTime="2.978832244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:52.306136399 +0000 UTC m=+323.032455348" watchObservedRunningTime="2026-09-10 16:22:52.309101321 +0000 UTC m=+323.035420251" Sep 10 16:22:52.810120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:52.810090 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:53.491808 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.491773 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-575575bffc-q55pv"] Sep 10 16:22:53.496920 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.496896 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.502847 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.502820 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 16:22:53.503017 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.502866 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 16:22:53.503017 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.502920 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qsxzr\"" Sep 10 16:22:53.503017 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.502952 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 16:22:53.510826 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.510801 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575575bffc-q55pv"] Sep 10 16:22:53.683388 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.683346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-manager-config\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.683388 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.683391 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6chf\" (UniqueName: \"kubernetes.io/projected/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-kube-api-access-d6chf\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.683609 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.683490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-cert\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.683609 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.683549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-metrics-cert\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.784566 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.784522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-metrics-cert\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.784767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.784583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-manager-config\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.784767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.784615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6chf\" (UniqueName: \"kubernetes.io/projected/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-kube-api-access-d6chf\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.784767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.784700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-cert\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.785672 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.785641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-manager-config\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.787794 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.787758 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-metrics-cert\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.788255 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.788233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-cert\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.797288 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.797263 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6chf\" (UniqueName: \"kubernetes.io/projected/ccc2300e-db24-4ad7-9d65-99eba08bbcc4-kube-api-access-d6chf\") pod \"lws-controller-manager-575575bffc-q55pv\" (UID: \"ccc2300e-db24-4ad7-9d65-99eba08bbcc4\") " pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.806134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.806084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:53.817966 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:53.817933 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:54.096243 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.096197 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575575bffc-q55pv"] Sep 10 16:22:54.099548 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:22:54.099518 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc2300e_db24_4ad7_9d65_99eba08bbcc4.slice/crio-1b26064eaddcd63ff0b019ba93d9ba6aa955b533d91d108074120ba90f1accea WatchSource:0}: Error finding container 1b26064eaddcd63ff0b019ba93d9ba6aa955b533d91d108074120ba90f1accea: Status 404 returned error can't find the container with id 1b26064eaddcd63ff0b019ba93d9ba6aa955b533d91d108074120ba90f1accea Sep 10 16:22:54.283032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.282993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" event={"ID":"ccc2300e-db24-4ad7-9d65-99eba08bbcc4","Type":"ContainerStarted","Data":"1b26064eaddcd63ff0b019ba93d9ba6aa955b533d91d108074120ba90f1accea"} Sep 10 16:22:54.283266 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.283244 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:54.284610 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.284592 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz" Sep 10 16:22:54.375936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.375858 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc"] Sep 10 16:22:54.378880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.378859 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.381829 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.381807 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 16:22:54.382549 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.382524 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 16:22:54.383104 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.383050 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 16:22:54.383256 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.383237 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6hh52\"" Sep 10 16:22:54.397105 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.397080 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc"] Sep 10 16:22:54.493322 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.493279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.493322 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.493323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.493789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.493350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.493789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.493431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.493789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.493470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ffffbd-1637-49ae-9e51-f7398e7add88-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.493789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.493487 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.493789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.493546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4856\" (UniqueName: \"kubernetes.io/projected/31ffffbd-1637-49ae-9e51-f7398e7add88-kube-api-access-q4856\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.593915 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.593877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ffffbd-1637-49ae-9e51-f7398e7add88-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.593915 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.593917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.594202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.593946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4856\" (UniqueName: \"kubernetes.io/projected/31ffffbd-1637-49ae-9e51-f7398e7add88-kube-api-access-q4856\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.594202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.594073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.594202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.594096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.594202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.594118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.594401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.594202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.594960 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.594933 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.596491 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.596460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ffffbd-1637-49ae-9e51-f7398e7add88-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.596602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.596533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.596602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.596586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.596685 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.596627 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.603943 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.603918 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ffffbd-1637-49ae-9e51-f7398e7add88-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.604116 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.604099 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4856\" (UniqueName: \"kubernetes.io/projected/31ffffbd-1637-49ae-9e51-f7398e7add88-kube-api-access-q4856\") pod \"istiod-openshift-gateway-5677ccb49b-p5cjc\" (UID: \"31ffffbd-1637-49ae-9e51-f7398e7add88\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:54.690375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:54.690288 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:55.007184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:55.007112 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc"] Sep 10 16:22:55.012508 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:22:55.012469 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ffffbd_1637_49ae_9e51_f7398e7add88.slice/crio-a648858a85bfd8ad7046f6bbe18ad09fbebf110d9aade1294a41c717af9ea23b WatchSource:0}: Error finding container a648858a85bfd8ad7046f6bbe18ad09fbebf110d9aade1294a41c717af9ea23b: Status 404 returned error can't find the container with id a648858a85bfd8ad7046f6bbe18ad09fbebf110d9aade1294a41c717af9ea23b Sep 10 16:22:55.289510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:55.289418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" event={"ID":"31ffffbd-1637-49ae-9e51-f7398e7add88","Type":"ContainerStarted","Data":"a648858a85bfd8ad7046f6bbe18ad09fbebf110d9aade1294a41c717af9ea23b"} Sep 10 16:22:56.295936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:56.295900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" event={"ID":"ccc2300e-db24-4ad7-9d65-99eba08bbcc4","Type":"ContainerStarted","Data":"6834c5ed0b1039596ce877c631c97f6e6e05e83ebaf0954df378c216dd89f551"} Sep 10 16:22:56.332414 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:56.327977 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" podStartSLOduration=1.838489681 podStartE2EDuration="3.327951437s" totalImagesPullingTime="1.489461756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:56.325441224 +0000 UTC m=+327.051760157" watchObservedRunningTime="2026-09-10 16:22:56.327951437 +0000 UTC m=+327.054270368" Sep 10 16:22:56.469315 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:56.469280 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:56.660627 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:56.660540 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:56.859547 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:56.859507 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:57.074446 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:57.074410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:57.302267 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:57.302236 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:22:57.886674 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:57.886351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:58.023934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:58.023894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:22:58.134285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:58.134248 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:22:58.134413 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:58.134326 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:22:58.309676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:58.309637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" event={"ID":"31ffffbd-1637-49ae-9e51-f7398e7add88","Type":"ContainerStarted","Data":"553c9f4bda58435b5a7d2384169789b7e7c2f1159b10abe12607d1aa24a954d0"} Sep 10 16:22:58.310151 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:58.309769 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:22:58.347961 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:58.347900 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" podStartSLOduration=1.229484475 podStartE2EDuration="4.347881855s" totalImagesPullingTime="3.11839738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:22:58.344555974 +0000 UTC m=+329.070874929" watchObservedRunningTime="2026-09-10 16:22:58.347881855 +0000 UTC m=+329.074200785" Sep 10 16:22:59.316500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:22:59.316471 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5cjc" Sep 10 16:23:08.312652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:08.312613 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-575575bffc-q55pv" Sep 10 16:23:28.739425 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:28.739344 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:28.814370 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:28.814322 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:40.178188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:40.178156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:40.279466 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:40.279424 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:40.381710 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:40.381678 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:46.533798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:46.533760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:46.641305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:46.641260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:59.443254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:59.443214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:23:59.543668 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:23:59.543639 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:01.934275 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:01.934237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:04.766262 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:04.766221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:06.409534 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:06.409496 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:06.925962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:06.925933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:07.329757 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:07.329729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:07.582596 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:07.582515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:07.777625 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:07.777596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:08.413027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:08.412990 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:08.515030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:08.514996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:10.912116 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:10.912073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:11.343976 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:11.343938 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:12.464761 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:12.464724 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:12.576396 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:12.576363 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:15.343023 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:15.342981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:15.447149 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:15.447115 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:18.244385 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:18.244339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:18.736092 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:18.736029 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:18.836324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:18.836292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:23.535585 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:23.535514 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:23.613863 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:23.613825 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:24.142023 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:24.141993 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:24.243741 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:24.243710 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:24.743611 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:24.743574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:24.822482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:24.822442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:29.159874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.159833 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57"] Sep 10 16:24:29.162379 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.162357 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:29.165324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.165303 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 16:24:29.165804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.165784 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sq4hz\"" Sep 10 16:24:29.165896 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.165789 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 16:24:29.181333 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.181301 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57"] Sep 10 16:24:29.281621 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.281584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgnp\" (UniqueName: \"kubernetes.io/projected/d1998dd4-7015-4955-a5c4-b67934dc917e-kube-api-access-fjgnp\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" (UID: \"d1998dd4-7015-4955-a5c4-b67934dc917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:29.281792 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.281631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1998dd4-7015-4955-a5c4-b67934dc917e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" (UID: \"d1998dd4-7015-4955-a5c4-b67934dc917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:29.383230 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.383172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgnp\" (UniqueName: \"kubernetes.io/projected/d1998dd4-7015-4955-a5c4-b67934dc917e-kube-api-access-fjgnp\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" (UID: \"d1998dd4-7015-4955-a5c4-b67934dc917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:29.383441 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.383245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1998dd4-7015-4955-a5c4-b67934dc917e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" (UID: \"d1998dd4-7015-4955-a5c4-b67934dc917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:29.383690 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.383667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1998dd4-7015-4955-a5c4-b67934dc917e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" (UID: \"d1998dd4-7015-4955-a5c4-b67934dc917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:29.396205 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.396179 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgnp\" (UniqueName: \"kubernetes.io/projected/d1998dd4-7015-4955-a5c4-b67934dc917e-kube-api-access-fjgnp\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" (UID: \"d1998dd4-7015-4955-a5c4-b67934dc917e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:29.472471 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.472387 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:29.517705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.517668 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57"] Sep 10 16:24:29.529942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:29.529900 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57"] Sep 10 16:24:29.871995 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:24:29.871960 2562 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 10 16:24:29.871995 ip-10-0-139-24 kubenswrapper[2562]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-65z57_kuadrant-system_d1998dd4-7015-4955-a5c4-b67934dc917e_0(7c9bbce217a6313bcce9a3388d50dd22e70a6ddba47d3e21dc7cc1f7a7f600a0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-65z57 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7c9bbce217a6313bcce9a3388d50dd22e70a6ddba47d3e21dc7cc1f7a7f600a0" Netns:"/var/run/netns/469a8812-b3dd-4a1d-a19d-f496c68bee23" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-65z57;K8S_POD_INFRA_CONTAINER_ID=7c9bbce217a6313bcce9a3388d50dd22e70a6ddba47d3e21dc7cc1f7a7f600a0;K8S_POD_UID=d1998dd4-7015-4955-a5c4-b67934dc917e" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57/d1998dd4-7015-4955-a5c4-b67934dc917e:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57 7c9bbce217a6313bcce9a3388d50dd22e70a6ddba47d3e21dc7cc1f7a7f600a0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:1c [10.134.0.28/23] Sep 10 16:24:29.871995 ip-10-0-139-24 kubenswrapper[2562]: ' Sep 10 16:24:29.871995 ip-10-0-139-24 kubenswrapper[2562]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 10 16:24:29.871995 ip-10-0-139-24 kubenswrapper[2562]: > Sep 10 16:24:29.872226 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:24:29.872026 2562 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 10 16:24:29.872226 ip-10-0-139-24 kubenswrapper[2562]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-65z57_kuadrant-system_d1998dd4-7015-4955-a5c4-b67934dc917e_0(7c9bbce217a6313bcce9a3388d50dd22e70a6ddba47d3e21dc7cc1f7a7f600a0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-65z57 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7c9bbce217a6313bcce9a3388d50dd22e70a6ddba47d3e21dc7cc1f7a7f600a0" Netns:"/var/run/netns/469a8812-b3dd-4a1d-a19d-f496c68bee23" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-65z57;K8S_POD_INFRA_CONTAINER_ID=7c9bbce217a6313bcce9a3388d50dd22e70a6ddba47d3e21dc7cc1f7a7f600a0;K8S_POD_UID=d1998dd4-7015-4955-a5c4-b67934dc917e" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57/d1998dd4-7015-4955-a5c4-b67934dc917e:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57 7c9bbce217a6313bcce9a3388d50dd22e70a6ddba47d3e21dc7cc1f7a7f600a0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:1c [10.134.0.28/23] Sep 10 16:24:29.872226 ip-10-0-139-24 kubenswrapper[2562]: ' Sep 10 16:24:29.872226 ip-10-0-139-24 kubenswrapper[2562]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 10 16:24:29.872226 ip-10-0-139-24 kubenswrapper[2562]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:30.741323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:30.741293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:30.747748 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:30.747725 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:30.750573 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:24:30.750547 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" is forbidden: User \"system:node:ip-10-0-139-24.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-24.ec2.internal' and this object" podUID="d1998dd4-7015-4955-a5c4-b67934dc917e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:30.795024 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:30.794986 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjgnp\" (UniqueName: \"kubernetes.io/projected/d1998dd4-7015-4955-a5c4-b67934dc917e-kube-api-access-fjgnp\") pod \"d1998dd4-7015-4955-a5c4-b67934dc917e\" (UID: \"d1998dd4-7015-4955-a5c4-b67934dc917e\") " Sep 10 16:24:30.795239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:30.795089 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1998dd4-7015-4955-a5c4-b67934dc917e-extensions-socket-volume\") pod \"d1998dd4-7015-4955-a5c4-b67934dc917e\" (UID: \"d1998dd4-7015-4955-a5c4-b67934dc917e\") " Sep 10 16:24:30.795491 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:30.795350 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1998dd4-7015-4955-a5c4-b67934dc917e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d1998dd4-7015-4955-a5c4-b67934dc917e" (UID: "d1998dd4-7015-4955-a5c4-b67934dc917e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:24:30.797080 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:30.797055 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1998dd4-7015-4955-a5c4-b67934dc917e-kube-api-access-fjgnp" (OuterVolumeSpecName: "kube-api-access-fjgnp") pod "d1998dd4-7015-4955-a5c4-b67934dc917e" (UID: "d1998dd4-7015-4955-a5c4-b67934dc917e"). InnerVolumeSpecName "kube-api-access-fjgnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:24:30.895859 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:30.895815 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1998dd4-7015-4955-a5c4-b67934dc917e-extensions-socket-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:24:30.895859 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:30.895852 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjgnp\" (UniqueName: \"kubernetes.io/projected/d1998dd4-7015-4955-a5c4-b67934dc917e-kube-api-access-fjgnp\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:24:31.746271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:31.746238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:31.750118 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:24:31.750081 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" is forbidden: User \"system:node:ip-10-0-139-24.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-24.ec2.internal' and this object" podUID="d1998dd4-7015-4955-a5c4-b67934dc917e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:31.755738 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:24:31.755707 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-65z57\" is forbidden: User \"system:node:ip-10-0-139-24.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-24.ec2.internal' and this object" podUID="d1998dd4-7015-4955-a5c4-b67934dc917e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-65z57" Sep 10 16:24:31.757840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:31.757818 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1998dd4-7015-4955-a5c4-b67934dc917e" path="/var/lib/kubelet/pods/d1998dd4-7015-4955-a5c4-b67934dc917e/volumes" Sep 10 16:24:33.575577 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:33.575537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:33.678095 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:33.678023 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:33.956099 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:33.955957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:34.061793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:34.061763 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:38.551397 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:38.551348 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:38.664010 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:38.663966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:42.164529 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:42.164491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:42.296871 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:42.296837 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:46.289976 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:46.289904 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:46.454833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:46.454802 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:46.576197 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:46.576091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:55.019019 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:55.018905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:55.130498 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:55.130466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:57.940432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.940389 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq"] Sep 10 16:24:57.944245 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.944221 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945075 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945110 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945178 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945234 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945258 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2c9\" (UniqueName: \"kubernetes.io/projected/9ebc2bba-8023-4022-8a30-c5ecae72b843-kube-api-access-jn2c9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.945600 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.945403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebc2bba-8023-4022-8a30-c5ecae72b843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:57.947945 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.947914 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xnlcl\"" Sep 10 16:24:57.961230 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.961200 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq"] Sep 10 16:24:57.981340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:57.981304 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:58.045989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.045953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2c9\" (UniqueName: \"kubernetes.io/projected/9ebc2bba-8023-4022-8a30-c5ecae72b843-kube-api-access-jn2c9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebc2bba-8023-4022-8a30-c5ecae72b843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046112 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046210 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046657 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046657 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046771 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046653 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.046831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.046787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebc2bba-8023-4022-8a30-c5ecae72b843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.047147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.047123 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.047502 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.047475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.050000 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.049979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.050129 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.050054 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.055593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.055569 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebc2bba-8023-4022-8a30-c5ecae72b843-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.055702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.055648 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2c9\" (UniqueName: \"kubernetes.io/projected/9ebc2bba-8023-4022-8a30-c5ecae72b843-kube-api-access-jn2c9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v25lq\" (UID: \"9ebc2bba-8023-4022-8a30-c5ecae72b843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.182209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.182174 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:24:58.254806 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.254769 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:58.533667 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.533641 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq"] Sep 10 16:24:58.536067 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:24:58.536014 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebc2bba_8023_4022_8a30_c5ecae72b843.slice/crio-5f9d9025f2a50c3fafd7a10b7379e744a80ee20e860205f9ec767db7b6bb2771 WatchSource:0}: Error finding container 5f9d9025f2a50c3fafd7a10b7379e744a80ee20e860205f9ec767db7b6bb2771: Status 404 returned error can't find the container with id 5f9d9025f2a50c3fafd7a10b7379e744a80ee20e860205f9ec767db7b6bb2771 Sep 10 16:24:58.538112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.538078 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:24:58.538211 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.538150 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:24:58.538211 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.538203 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:24:58.875417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.875321 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" event={"ID":"9ebc2bba-8023-4022-8a30-c5ecae72b843","Type":"ContainerStarted","Data":"3b4d2e87c48d81eda121bc1754f1e38aadd4b9e413b3e6db21abdffbf5a1c6c0"} Sep 10 16:24:58.875417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.875377 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" event={"ID":"9ebc2bba-8023-4022-8a30-c5ecae72b843","Type":"ContainerStarted","Data":"5f9d9025f2a50c3fafd7a10b7379e744a80ee20e860205f9ec767db7b6bb2771"} Sep 10 16:24:58.903546 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:58.903489 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" podStartSLOduration=1.9034734370000002 podStartE2EDuration="1.903473437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:24:58.902190309 +0000 UTC m=+449.628509263" watchObservedRunningTime="2026-09-10 16:24:58.903473437 +0000 UTC m=+449.629792367" Sep 10 16:24:59.255726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:59.255688 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:59.260679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:59.260655 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:59.879651 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:59.879620 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:24:59.880551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:24:59.880531 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:25:12.558563 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.558528 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:12.653165 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.653132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:12.787429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.787394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:12.900859 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.900774 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:25:12.904635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.904611 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:12.907986 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.907963 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 16:25:12.908092 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.908011 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 16:25:12.909235 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.909211 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 16:25:12.909235 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.909222 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qnplg\"" Sep 10 16:25:12.916102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.916079 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:25:12.954118 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.954084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:25:12.983985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.983946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptj4x\" (UniqueName: \"kubernetes.io/projected/f9d37e27-89a2-4009-b1be-9a62058cb899-kube-api-access-ptj4x\") pod \"limitador-limitador-84994f7ffc-kgq7c\" (UID: \"f9d37e27-89a2-4009-b1be-9a62058cb899\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:12.984189 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:12.984094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f9d37e27-89a2-4009-b1be-9a62058cb899-config-file\") pod \"limitador-limitador-84994f7ffc-kgq7c\" (UID: \"f9d37e27-89a2-4009-b1be-9a62058cb899\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:13.084832 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.084795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f9d37e27-89a2-4009-b1be-9a62058cb899-config-file\") pod \"limitador-limitador-84994f7ffc-kgq7c\" (UID: \"f9d37e27-89a2-4009-b1be-9a62058cb899\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:13.085026 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.084867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptj4x\" (UniqueName: \"kubernetes.io/projected/f9d37e27-89a2-4009-b1be-9a62058cb899-kube-api-access-ptj4x\") pod \"limitador-limitador-84994f7ffc-kgq7c\" (UID: \"f9d37e27-89a2-4009-b1be-9a62058cb899\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:13.085551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.085524 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f9d37e27-89a2-4009-b1be-9a62058cb899-config-file\") pod \"limitador-limitador-84994f7ffc-kgq7c\" (UID: \"f9d37e27-89a2-4009-b1be-9a62058cb899\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:13.094537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.094515 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptj4x\" (UniqueName: \"kubernetes.io/projected/f9d37e27-89a2-4009-b1be-9a62058cb899-kube-api-access-ptj4x\") pod \"limitador-limitador-84994f7ffc-kgq7c\" (UID: \"f9d37e27-89a2-4009-b1be-9a62058cb899\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:13.216629 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.216531 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:13.432358 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.432321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:13.493817 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.493789 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:25:13.496245 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:25:13.496219 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d37e27_89a2_4009_b1be_9a62058cb899.slice/crio-25207d5f3d185f7f1704213fdb9753ef92178efa364468c44a94b5f6266b2777 WatchSource:0}: Error finding container 25207d5f3d185f7f1704213fdb9753ef92178efa364468c44a94b5f6266b2777: Status 404 returned error can't find the container with id 25207d5f3d185f7f1704213fdb9753ef92178efa364468c44a94b5f6266b2777 Sep 10 16:25:13.551178 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.551144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:13.950474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:13.950440 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" event={"ID":"f9d37e27-89a2-4009-b1be-9a62058cb899","Type":"ContainerStarted","Data":"25207d5f3d185f7f1704213fdb9753ef92178efa364468c44a94b5f6266b2777"} Sep 10 16:25:15.961738 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:15.961694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" event={"ID":"f9d37e27-89a2-4009-b1be-9a62058cb899","Type":"ContainerStarted","Data":"716b6cabfb68af204ada2e1039e80a0a73380e51402632dd08f2ce415657b05d"} Sep 10 16:25:15.962183 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:15.961802 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:15.983082 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:15.983016 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" podStartSLOduration=1.705411832 podStartE2EDuration="3.982997845s" totalImagesPullingTime="2.277586013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:25:15.982101469 +0000 UTC m=+466.708420410" watchObservedRunningTime="2026-09-10 16:25:15.982997845 +0000 UTC m=+466.709316807" Sep 10 16:25:16.145449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:16.145332 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:16.215696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:16.215656 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:21.885232 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:21.885202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:21.983259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:21.983226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:26.973747 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:26.968434 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-kgq7c" Sep 10 16:25:31.433582 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:31.433523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:31.543833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:31.543796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:31.989713 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:31.989675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:42.116896 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:42.116863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:42.795780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:42.795749 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:48.626527 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:48.626494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:48.723238 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:48.723203 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:48.825180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:48.825147 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:48.940361 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:48.940260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:49.016317 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:49.016275 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:52.211005 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:52.210958 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:25:52.307599 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:25:52.307565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:16.195433 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:16.195399 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:19.617399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:19.617345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:25.344516 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:25.344427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:25.519366 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:25.519321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:26.789355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:26.789319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:26.897768 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:26.897736 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:27.004160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:27.004124 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:27.352381 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:27.352340 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:27.473749 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:27.473714 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:27.577639 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:27.577600 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:26:27.888342 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:27.888308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:37.346295 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:37.346261 2562 generic.go:356] "Generic (PLEG): container finished" podID="9ebc2bba-8023-4022-8a30-c5ecae72b843" containerID="3b4d2e87c48d81eda121bc1754f1e38aadd4b9e413b3e6db21abdffbf5a1c6c0" exitCode=0 Sep 10 16:26:37.346721 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:37.346318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" event={"ID":"9ebc2bba-8023-4022-8a30-c5ecae72b843","Type":"ContainerDied","Data":"3b4d2e87c48d81eda121bc1754f1e38aadd4b9e413b3e6db21abdffbf5a1c6c0"} Sep 10 16:26:37.346777 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:37.346761 2562 scope.go:117] "RemoveContainer" containerID="3b4d2e87c48d81eda121bc1754f1e38aadd4b9e413b3e6db21abdffbf5a1c6c0" Sep 10 16:26:37.347924 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:37.347893 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:26:37.347988 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:37.347967 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:26:37.348023 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:37.347999 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:26:38.375052 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:38.375000 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" event={"ID":"9ebc2bba-8023-4022-8a30-c5ecae72b843","Type":"ContainerStarted","Data":"056a9585d64bb0a634d0a2e26b01a8e7330d03a0498560601c259f8b59dc54d8"} Sep 10 16:26:39.138656 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:39.138620 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:39.239466 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:39.239428 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:39.255387 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:39.255339 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:26:39.255864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:39.255825 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v25lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 16:26:39.255979 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:39.255897 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" podUID="9ebc2bba-8023-4022-8a30-c5ecae72b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 16:26:40.255555 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:40.255498 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v25lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 16:26:40.256058 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:40.255595 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" podUID="9ebc2bba-8023-4022-8a30-c5ecae72b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 16:26:41.256306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:41.256266 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v25lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 16:26:41.257318 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:41.256879 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" podUID="9ebc2bba-8023-4022-8a30-c5ecae72b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 16:26:42.256650 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:42.256186 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v25lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 16:26:42.256650 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:42.256260 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" podUID="9ebc2bba-8023-4022-8a30-c5ecae72b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 16:26:43.256000 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:43.255958 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v25lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 16:26:43.256318 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:43.256286 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" podUID="9ebc2bba-8023-4022-8a30-c5ecae72b843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 16:26:44.259442 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:44.259410 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:26:44.259825 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:44.259803 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:26:44.260611 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:44.260592 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v25lq" Sep 10 16:26:45.046342 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:45.046304 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:45.556523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:45.556482 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:26:55.084769 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:55.084733 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:55.288999 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:55.288966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:56.033948 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:56.033913 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:26:59.871396 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:59.871360 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:26:59.972809 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:26:59.972772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:07.322159 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:07.322122 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:10.277108 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:10.277066 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:11.357048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:11.357007 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:27:15.760221 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.760189 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5"] Sep 10 16:27:15.762998 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.762972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:15.773171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.773144 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 16:27:15.773306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.773177 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 16:27:15.773306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.773177 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h86xf\"" Sep 10 16:27:15.773306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.773177 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 16:27:15.783419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.783387 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5"] Sep 10 16:27:15.824516 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.824476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:15.928843 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.928806 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:15.929032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.928922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:15.929032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.928987 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:15.929032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.929021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16892c0d-d99a-47e9-9164-47144a4de1b5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:15.929264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.929116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfq2t\" (UniqueName: \"kubernetes.io/projected/16892c0d-d99a-47e9-9164-47144a4de1b5-kube-api-access-wfq2t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:15.929264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:15.929139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.029927 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.029832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.029927 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.029876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16892c0d-d99a-47e9-9164-47144a4de1b5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.029927 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.029913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfq2t\" (UniqueName: \"kubernetes.io/projected/16892c0d-d99a-47e9-9164-47144a4de1b5-kube-api-access-wfq2t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.030273 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.029935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.030273 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.029962 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.030273 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.030014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.030414 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.030344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.030414 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.030391 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.030494 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.030464 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.032373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.032352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16892c0d-d99a-47e9-9164-47144a4de1b5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.032536 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.032518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16892c0d-d99a-47e9-9164-47144a4de1b5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.042341 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.042315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfq2t\" (UniqueName: \"kubernetes.io/projected/16892c0d-d99a-47e9-9164-47144a4de1b5-kube-api-access-wfq2t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-96tt5\" (UID: \"16892c0d-d99a-47e9-9164-47144a4de1b5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.073426 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.073393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:16.351951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.351925 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5"] Sep 10 16:27:16.354169 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:27:16.354140 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16892c0d_d99a_47e9_9164_47144a4de1b5.slice/crio-dc7b7c20fbd5592fbeaad9f72a602134763d3b1c02e6b03dcb36d1bbb2a9e65f WatchSource:0}: Error finding container dc7b7c20fbd5592fbeaad9f72a602134763d3b1c02e6b03dcb36d1bbb2a9e65f: Status 404 returned error can't find the container with id dc7b7c20fbd5592fbeaad9f72a602134763d3b1c02e6b03dcb36d1bbb2a9e65f Sep 10 16:27:16.559057 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:16.559014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" event={"ID":"16892c0d-d99a-47e9-9164-47144a4de1b5","Type":"ContainerStarted","Data":"dc7b7c20fbd5592fbeaad9f72a602134763d3b1c02e6b03dcb36d1bbb2a9e65f"} Sep 10 16:27:17.058225 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:17.058184 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:27:17.569207 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:17.569172 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" event={"ID":"16892c0d-d99a-47e9-9164-47144a4de1b5","Type":"ContainerStarted","Data":"a75c67186be5a4e9d3f1fcde011c728fc4cafe5db1836692baaad4bd86483aa3"} Sep 10 16:27:17.592592 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:17.592533 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" podStartSLOduration=1.6006304120000001 podStartE2EDuration="2.592515528s" totalImagesPullingTime="991.885116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:27:17.590996508 +0000 UTC m=+588.317315438" watchObservedRunningTime="2026-09-10 16:27:17.592515528 +0000 UTC m=+588.318834458" Sep 10 16:27:24.675804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:24.675761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:26.073964 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:26.073921 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:26.074457 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:26.073999 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:26.086619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:26.086589 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:26.242925 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:26.242892 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:27:26.624494 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:26.624457 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-96tt5" Sep 10 16:27:29.706951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:29.706925 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:27:29.707379 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:29.706926 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:27:39.540588 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.540554 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7"] Sep 10 16:27:39.543102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.543081 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.546340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.546309 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 16:27:39.554758 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.554733 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7"] Sep 10 16:27:39.576284 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.576254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:39.637243 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.637209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.637421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.637266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b760bdb8-b675-4ab5-b992-ac6b035a6d28-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.637421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.637297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv54k\" (UniqueName: \"kubernetes.io/projected/b760bdb8-b675-4ab5-b992-ac6b035a6d28-kube-api-access-gv54k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.637421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.637317 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.637544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.637451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.637544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.637499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.738670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.738627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.738858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.738682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.738858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.738703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.738858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.738752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b760bdb8-b675-4ab5-b992-ac6b035a6d28-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.738858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.738785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv54k\" (UniqueName: \"kubernetes.io/projected/b760bdb8-b675-4ab5-b992-ac6b035a6d28-kube-api-access-gv54k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.738858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.738815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.739182 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.739124 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.739182 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.739160 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.739281 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.739221 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.741148 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.741125 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b760bdb8-b675-4ab5-b992-ac6b035a6d28-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.741335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.741313 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b760bdb8-b675-4ab5-b992-ac6b035a6d28-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.749077 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.749030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv54k\" (UniqueName: \"kubernetes.io/projected/b760bdb8-b675-4ab5-b992-ac6b035a6d28-kube-api-access-gv54k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csjb7\" (UID: \"b760bdb8-b675-4ab5-b992-ac6b035a6d28\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:39.854282 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:39.854178 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:40.132139 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:40.132076 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7"] Sep 10 16:27:40.134512 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:27:40.134476 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb760bdb8_b675_4ab5_b992_ac6b035a6d28.slice/crio-9874aa1c6b788670143762383ad728cbabfb619031ca6619f0023bc954be50cc WatchSource:0}: Error finding container 9874aa1c6b788670143762383ad728cbabfb619031ca6619f0023bc954be50cc: Status 404 returned error can't find the container with id 9874aa1c6b788670143762383ad728cbabfb619031ca6619f0023bc954be50cc Sep 10 16:27:40.678835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:40.678798 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" event={"ID":"b760bdb8-b675-4ab5-b992-ac6b035a6d28","Type":"ContainerStarted","Data":"b99a85759b197f993cec3b52b670a7d161e2d62543dbf364d6b499189ff7f39b"} Sep 10 16:27:40.679251 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:40.678851 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" event={"ID":"b760bdb8-b675-4ab5-b992-ac6b035a6d28","Type":"ContainerStarted","Data":"9874aa1c6b788670143762383ad728cbabfb619031ca6619f0023bc954be50cc"} Sep 10 16:27:40.702032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:40.701977 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" podStartSLOduration=1.539335026 podStartE2EDuration="1.701959499s" totalImagesPullingTime="162.624473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:27:40.70115229 +0000 UTC m=+611.427471240" watchObservedRunningTime="2026-09-10 16:27:40.701959499 +0000 UTC m=+611.428278428" Sep 10 16:27:41.465921 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:41.465885 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:27:49.104349 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:49.104307 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:27:49.855293 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:49.855259 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:49.855466 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:49.855325 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:49.867840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:49.867808 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:50.741101 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:50.741073 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csjb7" Sep 10 16:27:51.547353 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:51.547315 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:27:59.608703 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:27:59.608665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:28:20.894421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:28:20.894384 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:04.032799 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:04.032763 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:04.123065 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:04.123009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:04.652278 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:04.652246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:04.742594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:04.742551 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:05.737334 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:05.737295 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:05.841135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:05.841096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:20.780697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:20.780659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:20.905941 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:20.905910 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:21.199214 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:21.199129 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:21.422835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:21.422798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:22.142491 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:22.142420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:22.241966 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:22.241929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:22.619553 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:22.619516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:25.157394 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:25.157358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:29:27.055756 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:27.055712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:27.177870 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:27.177839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:28.397708 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:28.397675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:28.498356 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:28.498321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:28.740900 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:28.740808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:28.901861 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:28.901832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:28.999452 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:28.999422 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:29.346722 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.346640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:29.621503 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.621413 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk"] Sep 10 16:29:29.624594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.624572 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.628103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.628077 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-2d818c-openshift-default-dockercfg-kxdg5\"" Sep 10 16:29:29.644300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.644270 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk"] Sep 10 16:29:29.672634 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.672593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:29.709453 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-podinfo\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.709637 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvp7\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-kube-api-access-4nvp7\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.709637 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-credential-socket\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.709637 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-socket\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.709764 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-data\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.709764 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-envoy\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.709764 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-certs\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.709864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istiod-ca-cert\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.709864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.709849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-token\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.766828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.766796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:29.811296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-socket\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811485 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-data\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811485 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-envoy\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811485 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-certs\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811485 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istiod-ca-cert\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811704 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-token\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811704 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811662 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-podinfo\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811704 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811683 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-socket\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvp7\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-kube-api-access-4nvp7\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811759 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-data\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-credential-socket\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.811999 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.811883 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-certs\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.812156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.812134 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-credential-socket\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.812248 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.812231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istiod-ca-cert\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.813753 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.813731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-envoy\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.814117 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.814098 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-podinfo\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.820937 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.820913 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-token\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.821024 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.820973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvp7\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-kube-api-access-4nvp7\") pod \"e2e-autores-2d818c-openshift-default-5856789fd-57jbk\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:29.939391 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.939292 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-2d818c-openshift-default-dockercfg-kxdg5\"" Sep 10 16:29:29.947000 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:29.946957 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:30.225230 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.225200 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk"] Sep 10 16:29:30.227478 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:29:30.227450 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a74b5c2_a1f0_40a4_bb2d_3a27390925a7.slice/crio-3c6ddf3ac2b12bd524bceaa4875297a272cc0bdd82b9f276f9a5554df919e1d2 WatchSource:0}: Error finding container 3c6ddf3ac2b12bd524bceaa4875297a272cc0bdd82b9f276f9a5554df919e1d2: Status 404 returned error can't find the container with id 3c6ddf3ac2b12bd524bceaa4875297a272cc0bdd82b9f276f9a5554df919e1d2 Sep 10 16:29:30.230050 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.230001 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:30.230123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.230105 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:30.230162 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.230137 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:30.361690 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.361653 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:30.461373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.461294 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:30.648321 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.648286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:30.862667 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.862633 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z"] Sep 10 16:29:30.865219 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.865195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.868541 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.868517 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-25d77f0b-openshift-default-dockercfg-rq2n9\"" Sep 10 16:29:30.879564 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.879533 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z"] Sep 10 16:29:30.898702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.898622 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:30.922727 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.922667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-credential-socket\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.922727 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.922730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-envoy\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.922960 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.922781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-socket\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.922960 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.922812 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9mg\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-kube-api-access-xk9mg\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.922960 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.922871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-data\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.922960 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.922952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istiod-ca-cert\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.923158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.922976 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-token\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.923158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.922997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-podinfo\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:30.923158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:30.923018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-certs\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.023898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.023859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-credential-socket\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.023898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.023899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-envoy\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024201 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.023931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-socket\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024201 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.023972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9mg\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-kube-api-access-xk9mg\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024201 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024024 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-data\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024201 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istiod-ca-cert\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024201 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-token\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-podinfo\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-certs\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024356 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-credential-socket\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024628 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-socket\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024664 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-certs\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.024733 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024694 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-data\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.025009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.024984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istiod-ca-cert\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.026551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.026529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-envoy\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.026889 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.026860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-podinfo\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.033156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.033129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-token\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.033620 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.033602 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9mg\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-kube-api-access-xk9mg\") pod \"e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.169537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.169451 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:31.176498 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.176463 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:31.228713 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.228662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" event={"ID":"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7","Type":"ContainerStarted","Data":"ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb"} Sep 10 16:29:31.228868 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.228735 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" event={"ID":"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7","Type":"ContainerStarted","Data":"3c6ddf3ac2b12bd524bceaa4875297a272cc0bdd82b9f276f9a5554df919e1d2"} Sep 10 16:29:31.253153 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.253092 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" podStartSLOduration=2.253075081 podStartE2EDuration="2.253075081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:31.251482642 +0000 UTC m=+721.977801572" watchObservedRunningTime="2026-09-10 16:29:31.253075081 +0000 UTC m=+721.979394010" Sep 10 16:29:31.460694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.460629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z"] Sep 10 16:29:31.462973 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:29:31.462942 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede1f6ef_2a8d_40e5_b6a0_83330a3bc667.slice/crio-4a72a67a40ffe50a83c5d3294dc3dba8f920d90531a89e89f08406085a1c44a5 WatchSource:0}: Error finding container 4a72a67a40ffe50a83c5d3294dc3dba8f920d90531a89e89f08406085a1c44a5: Status 404 returned error can't find the container with id 4a72a67a40ffe50a83c5d3294dc3dba8f920d90531a89e89f08406085a1c44a5 Sep 10 16:29:31.467122 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.465612 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:31.467122 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.465767 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:31.467122 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.465871 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:31.947818 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.947781 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:31.952803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:31.952777 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:32.235679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:32.235589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" event={"ID":"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667","Type":"ContainerStarted","Data":"4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806"} Sep 10 16:29:32.235679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:32.235651 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" event={"ID":"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667","Type":"ContainerStarted","Data":"4a72a67a40ffe50a83c5d3294dc3dba8f920d90531a89e89f08406085a1c44a5"} Sep 10 16:29:32.235902 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:32.235787 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:32.236766 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:32.236749 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:32.260569 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:32.260506 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podStartSLOduration=2.26048356 podStartE2EDuration="2.26048356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:32.259744078 +0000 UTC m=+722.986062998" watchObservedRunningTime="2026-09-10 16:29:32.26048356 +0000 UTC m=+722.986802508" Sep 10 16:29:32.522118 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:32.522079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:33.177090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.177028 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:33.181597 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.181563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:33.182645 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.182625 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:33.241239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.241210 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:33.242145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.242125 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:33.272528 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.272492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:33.544991 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.544957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:33.644842 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.644805 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:33.748304 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.748267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:33.845598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.845513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:33.945998 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:33.945968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:34.922665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:34.922631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:34.961828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:34.961793 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:29:35.023619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:35.023584 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:35.227524 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:35.227426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:35.616977 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:35.616943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:35.768366 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:35.768333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:36.095185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:36.095154 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:36.911884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:36.911849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:37.738308 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:37.738272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:37.835396 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:37.835354 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:38.127790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.127756 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:38.226713 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.226670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:38.455981 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.455890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:38.737243 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.737161 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:38.834586 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.834555 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:38.891427 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.891392 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j"] Sep 10 16:29:38.894361 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.894330 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:38.897373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.897350 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-25d77f0b\"" Sep 10 16:29:38.897543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.897526 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-25d77f0b-dockercfg-2mqcg\"" Sep 10 16:29:38.912718 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:38.912683 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j"] Sep 10 16:29:39.010287 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.010252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/4d4df9ea-9981-46c4-ba19-fdf6741d318d-kube-api-access-65qnd\") pod \"payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j\" (UID: \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\") " pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:39.010466 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.010321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4df9ea-9981-46c4-ba19-fdf6741d318d-plugins-config-volume\") pod \"payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j\" (UID: \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\") " pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:39.111913 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.111877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/4d4df9ea-9981-46c4-ba19-fdf6741d318d-kube-api-access-65qnd\") pod \"payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j\" (UID: \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\") " pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:39.112100 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.111963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4df9ea-9981-46c4-ba19-fdf6741d318d-plugins-config-volume\") pod \"payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j\" (UID: \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\") " pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:39.112661 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.112641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4df9ea-9981-46c4-ba19-fdf6741d318d-plugins-config-volume\") pod \"payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j\" (UID: \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\") " pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:39.122594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.122563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/4d4df9ea-9981-46c4-ba19-fdf6741d318d-kube-api-access-65qnd\") pod \"payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j\" (UID: \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\") " pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:39.207270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.207233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:39.469133 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.469015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:39.485747 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.485723 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j"] Sep 10 16:29:39.487754 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:29:39.487715 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4df9ea_9981_46c4_ba19_fdf6741d318d.slice/crio-d5c98914d158634712d9f0f2b45519bf0f111762c18578bd98960264e3faa575 WatchSource:0}: Error finding container d5c98914d158634712d9f0f2b45519bf0f111762c18578bd98960264e3faa575: Status 404 returned error can't find the container with id d5c98914d158634712d9f0f2b45519bf0f111762c18578bd98960264e3faa575 Sep 10 16:29:39.489630 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:39.489612 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:29:40.245932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:40.245796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:40.283748 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:40.283670 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" event={"ID":"4d4df9ea-9981-46c4-ba19-fdf6741d318d","Type":"ContainerStarted","Data":"d5c98914d158634712d9f0f2b45519bf0f111762c18578bd98960264e3faa575"} Sep 10 16:29:40.349096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:40.349025 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:41.240964 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.240933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:41.290358 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.290325 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" event={"ID":"4d4df9ea-9981-46c4-ba19-fdf6741d318d","Type":"ContainerStarted","Data":"08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9"} Sep 10 16:29:41.290763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.290420 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:41.332057 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.327364 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" podStartSLOduration=1.824152564 podStartE2EDuration="3.327340922s" totalImagesPullingTime="1.503188358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:41.317934646 +0000 UTC m=+732.044253586" watchObservedRunningTime="2026-09-10 16:29:41.327340922 +0000 UTC m=+732.053659854" Sep 10 16:29:41.382961 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.382911 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:41.498561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.498512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:41.641008 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.640918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:41.807646 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.807606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:41.919739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:41.919646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:42.147316 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.147278 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:42.529950 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.529914 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh"] Sep 10 16:29:42.532543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.532521 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.539044 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.539017 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9f6082-openshift-default-dockercfg-xff4d\"" Sep 10 16:29:42.557367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.557338 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh"] Sep 10 16:29:42.622400 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.622367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:42.648939 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.648904 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-credential-socket\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.649185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.648969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-data\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.649185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.649081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-envoy\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.649185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.649107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-podinfo\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.649185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.649134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-socket\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.649429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.649234 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr4ww\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-kube-api-access-lr4ww\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.649429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.649281 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-token\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.649429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.649320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dc34e2-1be0-49c6-88dc-fd83453b041f-istiod-ca-cert\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.649429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.649350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-certs\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750372 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-socket\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr4ww\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-kube-api-access-lr4ww\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-token\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dc34e2-1be0-49c6-88dc-fd83453b041f-istiod-ca-cert\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-certs\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-credential-socket\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-data\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-envoy\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-podinfo\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.750813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750799 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-socket\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.751013 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.750950 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-certs\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.751087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.751050 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-credential-socket\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.751134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.751104 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-data\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.751233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.751216 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dc34e2-1be0-49c6-88dc-fd83453b041f-istiod-ca-cert\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.752971 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.752945 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-envoy\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.753110 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.753070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-podinfo\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.770991 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.770966 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-token\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.772173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.772148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr4ww\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-kube-api-access-lr4ww\") pod \"e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.843355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.843272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:42.845579 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.845538 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:42.978483 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:42.978449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:43.132700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:43.132668 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh"] Sep 10 16:29:43.134192 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:29:43.134162 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dc34e2_1be0_49c6_88dc_fd83453b041f.slice/crio-1aa2e7a294f2a8f05704644a822a216b02f897e263bc1c48e23682f18258aa04 WatchSource:0}: Error finding container 1aa2e7a294f2a8f05704644a822a216b02f897e263bc1c48e23682f18258aa04: Status 404 returned error can't find the container with id 1aa2e7a294f2a8f05704644a822a216b02f897e263bc1c48e23682f18258aa04 Sep 10 16:29:43.136540 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:43.136508 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:43.136622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:43.136571 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:43.136622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:43.136600 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:43.304304 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:43.304257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" event={"ID":"14dc34e2-1be0-49c6-88dc-fd83453b041f","Type":"ContainerStarted","Data":"905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a"} Sep 10 16:29:43.304507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:43.304335 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" event={"ID":"14dc34e2-1be0-49c6-88dc-fd83453b041f","Type":"ContainerStarted","Data":"1aa2e7a294f2a8f05704644a822a216b02f897e263bc1c48e23682f18258aa04"} Sep 10 16:29:43.336921 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:43.336856 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" podStartSLOduration=1.336836366 podStartE2EDuration="1.336836366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:29:43.335851286 +0000 UTC m=+734.062170217" watchObservedRunningTime="2026-09-10 16:29:43.336836366 +0000 UTC m=+734.063155298" Sep 10 16:29:43.843770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:43.843718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:44.509392 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:44.509341 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:44.849327 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:44.849246 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:45.287196 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:45.287143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:45.315820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:45.315791 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:45.316816 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:45.316793 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:29:45.410887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:45.410849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:45.521420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:45.521390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:46.434750 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:46.434712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:46.734910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:46.734803 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:47.945074 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:47.945007 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:49.335458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:49.335418 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:49.452247 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:49.452206 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:49.549508 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:49.549472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:50.001835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:50.001802 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:50.104245 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:50.104210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:50.794125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:50.794086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:50.895686 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:50.895631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:51.762496 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:51.762437 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:52.298388 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:52.298354 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:29:52.670183 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:52.669977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:52.995231 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:52.995197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:53.525807 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:53.525728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:54.386316 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:54.386274 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:55.193909 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:55.193870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:55.876266 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:55.876215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:56.407270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:56.407184 2562 generic.go:356] "Generic (PLEG): container finished" podID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerID="4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806" exitCode=0 Sep 10 16:29:56.407270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:56.407225 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" event={"ID":"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667","Type":"ContainerDied","Data":"4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806"} Sep 10 16:29:56.407697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:56.407617 2562 scope.go:117] "RemoveContainer" containerID="4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806" Sep 10 16:29:56.408895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:56.408861 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:56.408997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:56.408942 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:56.408997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:56.408973 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:56.714523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:56.714485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:57.414365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:57.414333 2562 generic.go:356] "Generic (PLEG): container finished" podID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerID="ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb" exitCode=0 Sep 10 16:29:57.414806 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:57.414406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" event={"ID":"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7","Type":"ContainerDied","Data":"ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb"} Sep 10 16:29:57.414965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:57.414941 2562 scope.go:117] "RemoveContainer" containerID="ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb" Sep 10 16:29:57.416235 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:57.416191 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:57.416367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:57.416282 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:57.416367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:57.416331 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:29:57.416677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:57.416657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" event={"ID":"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667","Type":"ContainerStarted","Data":"d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff"} Sep 10 16:29:57.496964 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:57.496935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:58.150545 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:58.150497 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:29:58.177278 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:58.177235 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:29:58.177496 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:58.177373 2562 patch_prober.go:28] interesting pod/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 16:29:58.177496 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:58.177420 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 16:29:58.423951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:58.423864 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" event={"ID":"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7","Type":"ContainerStarted","Data":"6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898"} Sep 10 16:29:58.947562 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:58.947517 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:29:58.948880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:58.948846 2562 patch_prober.go:28] interesting pod/e2e-autores-2d818c-openshift-default-5856789fd-57jbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 16:29:58.949030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:58.948927 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 16:29:59.083475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.083437 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc"] Sep 10 16:29:59.087300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.087273 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" Sep 10 16:29:59.090510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.090484 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:29:59.090936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.090918 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:29:59.097269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.097236 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc"] Sep 10 16:29:59.157999 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.157957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d1a76b-b9a3-4c25-a689-5e75192cd892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc\" (UID: \"53d1a76b-b9a3-4c25-a689-5e75192cd892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" Sep 10 16:29:59.177506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.177463 2562 patch_prober.go:28] interesting pod/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 16:29:59.177787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.177756 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 16:29:59.259576 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.259535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d1a76b-b9a3-4c25-a689-5e75192cd892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc\" (UID: \"53d1a76b-b9a3-4c25-a689-5e75192cd892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" Sep 10 16:29:59.261011 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.260882 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d1a76b-b9a3-4c25-a689-5e75192cd892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc\" (UID: \"53d1a76b-b9a3-4c25-a689-5e75192cd892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" Sep 10 16:29:59.373389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.373304 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:29:59.403765 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.403715 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" Sep 10 16:29:59.808124 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.803124 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc"] Sep 10 16:29:59.947936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.947898 2562 patch_prober.go:28] interesting pod/e2e-autores-2d818c-openshift-default-5856789fd-57jbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 16:29:59.948249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:29:59.948219 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 16:30:00.148023 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.147918 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv"] Sep 10 16:30:00.160061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.157262 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g"] Sep 10 16:30:00.160061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.159546 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" Sep 10 16:30:00.164060 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.160851 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g" Sep 10 16:30:00.173058 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.172768 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g"] Sep 10 16:30:00.181059 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.178689 2562 patch_prober.go:28] interesting pod/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 16:30:00.181059 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.178805 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 16:30:00.181287 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.181063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv"] Sep 10 16:30:00.190566 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.190540 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" Sep 10 16:30:00.208080 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.204656 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g" Sep 10 16:30:00.237714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.237630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:00.462156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.462051 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" event={"ID":"53d1a76b-b9a3-4c25-a689-5e75192cd892","Type":"ContainerStarted","Data":"4425f6add90af08da3ad40495033c23c58b6b9707bb0608af6a8d84532f3e3cb"} Sep 10 16:30:00.911789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.907877 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g"] Sep 10 16:30:00.927069 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.925490 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv"] Sep 10 16:30:00.943687 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.940538 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:00.947165 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:30:00.946411 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85217ea0_360b_48c5_b35d_45413c837d9c.slice/crio-e37ca0113d5311e38ff195217b63e5ef008e57d6758158ba8d1e29dca5fda81f WatchSource:0}: Error finding container e37ca0113d5311e38ff195217b63e5ef008e57d6758158ba8d1e29dca5fda81f: Status 404 returned error can't find the container with id e37ca0113d5311e38ff195217b63e5ef008e57d6758158ba8d1e29dca5fda81f Sep 10 16:30:00.948140 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.948056 2562 patch_prober.go:28] interesting pod/e2e-autores-2d818c-openshift-default-5856789fd-57jbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 16:30:00.948140 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:00.948111 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 16:30:00.962367 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:30:00.961437 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060a2165_76e1_4968_98fd_6c9093414a83.slice/crio-8706db7d8a1aec49b085e465cfb4b929f295512e396c2a73c27c86b463540fc4 WatchSource:0}: Error finding container 8706db7d8a1aec49b085e465cfb4b929f295512e396c2a73c27c86b463540fc4: Status 404 returned error can't find the container with id 8706db7d8a1aec49b085e465cfb4b929f295512e396c2a73c27c86b463540fc4 Sep 10 16:30:01.178899 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:01.178797 2562 patch_prober.go:28] interesting pod/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 16:30:01.179200 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:01.179166 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 16:30:01.483301 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:01.483204 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g" event={"ID":"85217ea0-360b-48c5-b35d-45413c837d9c","Type":"ContainerStarted","Data":"e37ca0113d5311e38ff195217b63e5ef008e57d6758158ba8d1e29dca5fda81f"} Sep 10 16:30:01.490632 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:01.490541 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" event={"ID":"060a2165-76e1-4968-98fd-6c9093414a83","Type":"ContainerStarted","Data":"8706db7d8a1aec49b085e465cfb4b929f295512e396c2a73c27c86b463540fc4"} Sep 10 16:30:01.723604 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:01.723555 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:01.953833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:01.953341 2562 patch_prober.go:28] interesting pod/e2e-autores-2d818c-openshift-default-5856789fd-57jbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 16:30:01.953833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:01.953409 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 16:30:02.177945 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:02.177495 2562 patch_prober.go:28] interesting pod/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 16:30:02.177945 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:02.177560 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 16:30:02.236769 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:02.236668 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:30:02.515806 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:02.515753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:02.950125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:02.949518 2562 patch_prober.go:28] interesting pod/e2e-autores-2d818c-openshift-default-5856789fd-57jbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 16:30:02.950125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:02.949607 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 16:30:03.179162 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:03.178626 2562 patch_prober.go:28] interesting pod/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 16:30:03.179162 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:03.178695 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 16:30:03.246149 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:03.245149 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:30:03.420255 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:03.420177 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:03.950673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:03.948526 2562 patch_prober.go:28] interesting pod/e2e-autores-2d818c-openshift-default-5856789fd-57jbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 16:30:03.950673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:03.948591 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 16:30:04.068062 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:04.065559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:04.183765 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:04.183703 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:30:04.187624 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:04.187561 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:30:04.970061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:04.969997 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:30:04.971532 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:04.971468 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:30:04.996174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:04.996120 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:05.105957 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:05.105928 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:05.772512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:05.772483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:06.554565 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:06.554487 2562 generic.go:356] "Generic (PLEG): container finished" podID="85217ea0-360b-48c5-b35d-45413c837d9c" containerID="ce5c89c7e2da07bf4745d731fc3cd147e98b2ae69a30feb9cce17c82e7b1384e" exitCode=0 Sep 10 16:30:06.554804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:06.554668 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g" event={"ID":"85217ea0-360b-48c5-b35d-45413c837d9c","Type":"ContainerDied","Data":"ce5c89c7e2da07bf4745d731fc3cd147e98b2ae69a30feb9cce17c82e7b1384e"} Sep 10 16:30:06.556852 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:06.556818 2562 generic.go:356] "Generic (PLEG): container finished" podID="53d1a76b-b9a3-4c25-a689-5e75192cd892" containerID="d78c789e21bb4331eebf9444c18a1169a0768260124688fc80c10472cab6b10a" exitCode=0 Sep 10 16:30:06.556988 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:06.556952 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" event={"ID":"53d1a76b-b9a3-4c25-a689-5e75192cd892","Type":"ContainerDied","Data":"d78c789e21bb4331eebf9444c18a1169a0768260124688fc80c10472cab6b10a"} Sep 10 16:30:06.558898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:06.558872 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" event={"ID":"060a2165-76e1-4968-98fd-6c9093414a83","Type":"ContainerStarted","Data":"7e141378d90b88719e9c5c265e9d59cf8d536078ab624d426c23857e7d952170"} Sep 10 16:30:06.581531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:06.581493 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:06.612939 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:06.612879 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" podStartSLOduration=2.225511575 podStartE2EDuration="6.612861311s" totalImagesPullingTime="4.387349736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:06.611174507 +0000 UTC m=+757.337493438" watchObservedRunningTime="2026-09-10 16:30:06.612861311 +0000 UTC m=+757.339180240" Sep 10 16:30:07.440350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:07.440318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:07.566109 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:07.566072 2562 generic.go:356] "Generic (PLEG): container finished" podID="060a2165-76e1-4968-98fd-6c9093414a83" containerID="7e141378d90b88719e9c5c265e9d59cf8d536078ab624d426c23857e7d952170" exitCode=0 Sep 10 16:30:07.566293 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:07.566174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" event={"ID":"060a2165-76e1-4968-98fd-6c9093414a83","Type":"ContainerDied","Data":"7e141378d90b88719e9c5c265e9d59cf8d536078ab624d426c23857e7d952170"} Sep 10 16:30:07.723339 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:07.723306 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" Sep 10 16:30:07.759321 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:07.759292 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g" Sep 10 16:30:07.764395 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:07.764367 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d1a76b-b9a3-4c25-a689-5e75192cd892-maas-api-service-ca\") pod \"53d1a76b-b9a3-4c25-a689-5e75192cd892\" (UID: \"53d1a76b-b9a3-4c25-a689-5e75192cd892\") " Sep 10 16:30:07.764794 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:07.764766 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d1a76b-b9a3-4c25-a689-5e75192cd892-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53d1a76b-b9a3-4c25-a689-5e75192cd892" (UID: "53d1a76b-b9a3-4c25-a689-5e75192cd892"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:07.865523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:07.865478 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d1a76b-b9a3-4c25-a689-5e75192cd892-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:08.573366 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:08.573333 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g" Sep 10 16:30:08.575239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:08.573365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g" event={"ID":"85217ea0-360b-48c5-b35d-45413c837d9c","Type":"ContainerDied","Data":"e37ca0113d5311e38ff195217b63e5ef008e57d6758158ba8d1e29dca5fda81f"} Sep 10 16:30:08.575239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:08.573415 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37ca0113d5311e38ff195217b63e5ef008e57d6758158ba8d1e29dca5fda81f" Sep 10 16:30:08.575239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:08.574901 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" Sep 10 16:30:08.575239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:08.574902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc" event={"ID":"53d1a76b-b9a3-4c25-a689-5e75192cd892","Type":"ContainerDied","Data":"4425f6add90af08da3ad40495033c23c58b6b9707bb0608af6a8d84532f3e3cb"} Sep 10 16:30:08.575239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:08.574948 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4425f6add90af08da3ad40495033c23c58b6b9707bb0608af6a8d84532f3e3cb" Sep 10 16:30:08.731818 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:08.731786 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" Sep 10 16:30:09.581944 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.581913 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" Sep 10 16:30:09.582389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.581954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv" event={"ID":"060a2165-76e1-4968-98fd-6c9093414a83","Type":"ContainerDied","Data":"8706db7d8a1aec49b085e465cfb4b929f295512e396c2a73c27c86b463540fc4"} Sep 10 16:30:09.582389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.582004 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8706db7d8a1aec49b085e465cfb4b929f295512e396c2a73c27c86b463540fc4" Sep 10 16:30:09.583591 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.583564 2562 generic.go:356] "Generic (PLEG): container finished" podID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerID="905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a" exitCode=0 Sep 10 16:30:09.583700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.583627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" event={"ID":"14dc34e2-1be0-49c6-88dc-fd83453b041f","Type":"ContainerDied","Data":"905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a"} Sep 10 16:30:09.584064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.584031 2562 scope.go:117] "RemoveContainer" containerID="905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a" Sep 10 16:30:09.585231 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.585200 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:09.585333 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.585288 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:09.585395 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:09.585339 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:10.562843 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:10.562807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:10.591777 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:10.591599 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" event={"ID":"14dc34e2-1be0-49c6-88dc-fd83453b041f","Type":"ContainerStarted","Data":"51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a"} Sep 10 16:30:10.843879 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:10.843718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:30:10.844962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:10.844930 2562 patch_prober.go:28] interesting pod/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 16:30:10.845185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:10.845156 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 16:30:11.844627 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:11.844575 2562 patch_prober.go:28] interesting pod/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 16:30:11.845126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:11.844667 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 16:30:12.844633 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:12.844591 2562 patch_prober.go:28] interesting pod/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 16:30:12.845272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:12.845234 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 16:30:13.844226 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:13.844189 2562 patch_prober.go:28] interesting pod/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 16:30:13.844551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:13.844519 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 16:30:14.844262 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:14.844220 2562 patch_prober.go:28] interesting pod/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 16:30:14.844854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:14.844815 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 16:30:15.300296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.300266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:15.316373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.316344 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:30:15.847690 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.847658 2562 patch_prober.go:28] interesting pod/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 10 16:30:15.848335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.848301 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 10 16:30:15.875174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.875141 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv"] Sep 10 16:30:15.877545 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.877516 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-25d77f0b-29817630-kjlqv"] Sep 10 16:30:15.892993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.892965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:15.908691 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.908609 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j"] Sep 10 16:30:15.908925 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.908900 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" podUID="4d4df9ea-9981-46c4-ba19-fdf6741d318d" containerName="payload-processing" containerID="cri-o://08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9" gracePeriod=30 Sep 10 16:30:15.992997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:15.992968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:16.107197 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.107165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:16.123383 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.123346 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc"] Sep 10 16:30:16.126027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.125962 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25d77f0b-dgkfc"] Sep 10 16:30:16.199431 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.199399 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:16.204874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.204844 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:30:16.271443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.271404 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z"] Sep 10 16:30:16.271783 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.271748 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" containerID="cri-o://d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff" gracePeriod=30 Sep 10 16:30:16.291697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.291655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:16.358180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.358089 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4df9ea-9981-46c4-ba19-fdf6741d318d-plugins-config-volume\") pod \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\" (UID: \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\") " Sep 10 16:30:16.358338 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.358293 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/4d4df9ea-9981-46c4-ba19-fdf6741d318d-kube-api-access-65qnd\") pod \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\" (UID: \"4d4df9ea-9981-46c4-ba19-fdf6741d318d\") " Sep 10 16:30:16.358499 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.358469 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4df9ea-9981-46c4-ba19-fdf6741d318d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d4df9ea-9981-46c4-ba19-fdf6741d318d" (UID: "4d4df9ea-9981-46c4-ba19-fdf6741d318d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:16.358669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.358636 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4df9ea-9981-46c4-ba19-fdf6741d318d-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:16.361509 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.361472 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4df9ea-9981-46c4-ba19-fdf6741d318d-kube-api-access-65qnd" (OuterVolumeSpecName: "kube-api-access-65qnd") pod "4d4df9ea-9981-46c4-ba19-fdf6741d318d" (UID: "4d4df9ea-9981-46c4-ba19-fdf6741d318d"). InnerVolumeSpecName "kube-api-access-65qnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:16.459538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.459494 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/4d4df9ea-9981-46c4-ba19-fdf6741d318d-kube-api-access-65qnd\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:16.571164 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.571128 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:16.625088 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.624962 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d4df9ea-9981-46c4-ba19-fdf6741d318d" containerID="08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9" exitCode=0 Sep 10 16:30:16.625088 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.625006 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" Sep 10 16:30:16.625088 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.625071 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" event={"ID":"4d4df9ea-9981-46c4-ba19-fdf6741d318d","Type":"ContainerDied","Data":"08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9"} Sep 10 16:30:16.625377 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.625132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j" event={"ID":"4d4df9ea-9981-46c4-ba19-fdf6741d318d","Type":"ContainerDied","Data":"d5c98914d158634712d9f0f2b45519bf0f111762c18578bd98960264e3faa575"} Sep 10 16:30:16.625377 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.625160 2562 scope.go:117] "RemoveContainer" containerID="08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9" Sep 10 16:30:16.638342 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.638314 2562 scope.go:117] "RemoveContainer" containerID="08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9" Sep 10 16:30:16.639105 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:30:16.639077 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9\": container with ID starting with 08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9 not found: ID does not exist" containerID="08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9" Sep 10 16:30:16.639259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.639233 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9"} err="failed to get container status \"08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9\": rpc error: code = NotFound desc = could not find container \"08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9\": container with ID starting with 08df12fcb1cdb481a9adfb089203e569daf11c1f4f1d2ab3baae9a34a89789a9 not found: ID does not exist" Sep 10 16:30:16.660491 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.660460 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j"] Sep 10 16:30:16.670590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.670558 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25d77f0b-6fc547c65-srk8j"] Sep 10 16:30:16.705397 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.705360 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:16.803359 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.803327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:16.848369 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.848340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:30:16.849217 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.849198 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:30:16.933814 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:16.933716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:17.228323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228229 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6"] Sep 10 16:30:17.228838 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228814 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060a2165-76e1-4968-98fd-6c9093414a83" containerName="cleanup" Sep 10 16:30:17.228838 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228838 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="060a2165-76e1-4968-98fd-6c9093414a83" containerName="cleanup" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228851 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4df9ea-9981-46c4-ba19-fdf6741d318d" containerName="payload-processing" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228857 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4df9ea-9981-46c4-ba19-fdf6741d318d" containerName="payload-processing" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228872 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d1a76b-b9a3-4c25-a689-5e75192cd892" containerName="revoke-keys" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228878 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d1a76b-b9a3-4c25-a689-5e75192cd892" containerName="revoke-keys" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228898 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85217ea0-360b-48c5-b35d-45413c837d9c" containerName="cleanup" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228903 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="85217ea0-360b-48c5-b35d-45413c837d9c" containerName="cleanup" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228961 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53d1a76b-b9a3-4c25-a689-5e75192cd892" containerName="revoke-keys" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228972 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="85217ea0-360b-48c5-b35d-45413c837d9c" containerName="cleanup" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228979 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="060a2165-76e1-4968-98fd-6c9093414a83" containerName="cleanup" Sep 10 16:30:17.228994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.228987 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4df9ea-9981-46c4-ba19-fdf6741d318d" containerName="payload-processing" Sep 10 16:30:17.232024 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.232009 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.235702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.235685 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2238d044-gw-openshift-default-dockercfg-wkw4n\"" Sep 10 16:30:17.248056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.247750 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6"] Sep 10 16:30:17.250533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.250502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:17.368716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.368668 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-credential-socket\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.368946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.368725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-data\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.368946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.368830 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-certs\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.368946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.368864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1eedca-daae-4f5d-87a6-55997c3dc378-istiod-ca-cert\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.368946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.368902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-podinfo\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.368946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.368939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-socket\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.369313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.368999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-envoy\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.369313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.369061 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-token\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.369313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.369079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjssl\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-kube-api-access-sjssl\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.429161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.429124 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:17.470953 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.470911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-credential-socket\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.471183 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.470980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-data\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.471183 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.471026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-certs\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.471183 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.471064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1eedca-daae-4f5d-87a6-55997c3dc378-istiod-ca-cert\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.471183 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.471109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-podinfo\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.471183 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.471136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-socket\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.471547 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.471218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-envoy\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.471547 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.471264 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-token\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.471547 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.471288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjssl\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-kube-api-access-sjssl\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.472163 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.472132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-credential-socket\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.472430 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.472408 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-data\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.472662 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.472639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-certs\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.473761 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.473297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1eedca-daae-4f5d-87a6-55997c3dc378-istiod-ca-cert\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.473761 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.473529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-socket\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.476507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.476440 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-envoy\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.476857 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.476834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-podinfo\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.481421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.481342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjssl\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-kube-api-access-sjssl\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.482359 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.482343 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-token\") pod \"e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.542923 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.542881 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:17.696813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.696778 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:17.760639 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.760606 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060a2165-76e1-4968-98fd-6c9093414a83" path="/var/lib/kubelet/pods/060a2165-76e1-4968-98fd-6c9093414a83/volumes" Sep 10 16:30:17.761007 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.760990 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4df9ea-9981-46c4-ba19-fdf6741d318d" path="/var/lib/kubelet/pods/4d4df9ea-9981-46c4-ba19-fdf6741d318d/volumes" Sep 10 16:30:17.761370 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.761357 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53d1a76b-b9a3-4c25-a689-5e75192cd892" path="/var/lib/kubelet/pods/53d1a76b-b9a3-4c25-a689-5e75192cd892/volumes" Sep 10 16:30:17.832901 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.832874 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6"] Sep 10 16:30:17.835383 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:30:17.835353 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1eedca_daae_4f5d_87a6_55997c3dc378.slice/crio-8abedf353d61a179d567f43877045e1c3a72f4d6362da6d278db74c4e3c14841 WatchSource:0}: Error finding container 8abedf353d61a179d567f43877045e1c3a72f4d6362da6d278db74c4e3c14841: Status 404 returned error can't find the container with id 8abedf353d61a179d567f43877045e1c3a72f4d6362da6d278db74c4e3c14841 Sep 10 16:30:17.837770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.837733 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:17.837876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.837802 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:17.837876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:17.837841 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:18.638481 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:18.638447 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" event={"ID":"1f1eedca-daae-4f5d-87a6-55997c3dc378","Type":"ContainerStarted","Data":"03612f8233ef12a8fa4c12de6fcbce9ba5aaab7b7ce0e8b4ce1340479c4a2993"} Sep 10 16:30:18.638875 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:18.638497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" event={"ID":"1f1eedca-daae-4f5d-87a6-55997c3dc378","Type":"ContainerStarted","Data":"8abedf353d61a179d567f43877045e1c3a72f4d6362da6d278db74c4e3c14841"} Sep 10 16:30:18.676376 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:18.676313 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" podStartSLOduration=1.676290534 podStartE2EDuration="1.676290534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:18.675511813 +0000 UTC m=+769.401830743" watchObservedRunningTime="2026-09-10 16:30:18.676290534 +0000 UTC m=+769.402609466" Sep 10 16:30:18.724301 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:18.724266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:19.186215 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:19.186178 2562 patch_prober.go:28] interesting pod/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:30:19.186399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:19.186244 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:30:19.543120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:19.543079 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:19.549161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:19.549133 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:19.645530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:19.645503 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:19.646533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:19.646512 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:19.807690 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:19.807604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:20.211294 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:20.211207 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:20.310401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:20.310362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:20.755586 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:20.755551 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:20.855939 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:20.855906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:21.233753 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.233615 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:21.326508 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.326475 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:21.337479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.337447 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8"] Sep 10 16:30:21.339892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.339875 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:21.343031 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.343010 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2238d044\"" Sep 10 16:30:21.352573 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.352539 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8"] Sep 10 16:30:21.375855 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.375823 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g"] Sep 10 16:30:21.378830 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.378809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:21.381823 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.381794 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2238d044-dockercfg-287gj\"" Sep 10 16:30:21.391570 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.391542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g"] Sep 10 16:30:21.414293 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.414255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcj7\" (UniqueName: \"kubernetes.io/projected/e7bebef4-0b4d-4792-9851-920f01946466-kube-api-access-nzcj7\") pod \"payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8\" (UID: \"e7bebef4-0b4d-4792-9851-920f01946466\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:21.414486 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.414316 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bebef4-0b4d-4792-9851-920f01946466-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8\" (UID: \"e7bebef4-0b4d-4792-9851-920f01946466\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:21.515556 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.515521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bebef4-0b4d-4792-9851-920f01946466-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8\" (UID: \"e7bebef4-0b4d-4792-9851-920f01946466\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:21.515716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.515573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g\" (UID: \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\") " pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:21.515793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.515743 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs49n\" (UniqueName: \"kubernetes.io/projected/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-kube-api-access-vs49n\") pod \"payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g\" (UID: \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\") " pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:21.515877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.515854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcj7\" (UniqueName: \"kubernetes.io/projected/e7bebef4-0b4d-4792-9851-920f01946466-kube-api-access-nzcj7\") pod \"payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8\" (UID: \"e7bebef4-0b4d-4792-9851-920f01946466\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:21.516316 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.516300 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bebef4-0b4d-4792-9851-920f01946466-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8\" (UID: \"e7bebef4-0b4d-4792-9851-920f01946466\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:21.525360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.525327 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcj7\" (UniqueName: \"kubernetes.io/projected/e7bebef4-0b4d-4792-9851-920f01946466-kube-api-access-nzcj7\") pod \"payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8\" (UID: \"e7bebef4-0b4d-4792-9851-920f01946466\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:21.556479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.556450 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:30:21.616543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.616500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs49n\" (UniqueName: \"kubernetes.io/projected/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-kube-api-access-vs49n\") pod \"payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g\" (UID: \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\") " pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:21.616682 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.616600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g\" (UID: \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\") " pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:21.617348 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.617326 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g\" (UID: \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\") " pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:21.629027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.628997 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs49n\" (UniqueName: \"kubernetes.io/projected/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-kube-api-access-vs49n\") pod \"payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g\" (UID: \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\") " pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:21.656553 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.656526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:21.658226 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.658192 2562 generic.go:356] "Generic (PLEG): container finished" podID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerID="d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff" exitCode=0 Sep 10 16:30:21.658375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.658249 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" Sep 10 16:30:21.658375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.658292 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" event={"ID":"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667","Type":"ContainerDied","Data":"d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff"} Sep 10 16:30:21.658375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.658355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z" event={"ID":"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667","Type":"ContainerDied","Data":"4a72a67a40ffe50a83c5d3294dc3dba8f920d90531a89e89f08406085a1c44a5"} Sep 10 16:30:21.658527 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.658379 2562 scope.go:117] "RemoveContainer" containerID="d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff" Sep 10 16:30:21.667984 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.667958 2562 scope.go:117] "RemoveContainer" containerID="4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806" Sep 10 16:30:21.678747 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.678724 2562 scope.go:117] "RemoveContainer" containerID="d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff" Sep 10 16:30:21.679292 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:30:21.679266 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff\": container with ID starting with d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff not found: ID does not exist" containerID="d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff" Sep 10 16:30:21.679388 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.679304 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff"} err="failed to get container status \"d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff\": rpc error: code = NotFound desc = could not find container \"d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff\": container with ID starting with d198e612c8191b68745b58c288a146c0cae4c8c053e2b69823b0d57b7d0c22ff not found: ID does not exist" Sep 10 16:30:21.679388 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.679331 2562 scope.go:117] "RemoveContainer" containerID="4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806" Sep 10 16:30:21.679975 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:30:21.679946 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806\": container with ID starting with 4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806 not found: ID does not exist" containerID="4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806" Sep 10 16:30:21.680074 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.679976 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806"} err="failed to get container status \"4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806\": rpc error: code = NotFound desc = could not find container \"4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806\": container with ID starting with 4f671e5ceb5835c029d4bb652b33b550ea041ed056db7eb51a94e1dad983d806 not found: ID does not exist" Sep 10 16:30:21.691216 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.691182 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:21.710812 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.710758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:21.717647 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717368 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-data\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.717647 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717425 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-credential-socket\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.717647 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717457 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-socket\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.717647 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717493 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-token\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.717647 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717547 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istiod-ca-cert\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.717647 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717622 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-certs\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.718269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717660 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-data" (OuterVolumeSpecName: "istio-data") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:21.718269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717671 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk9mg\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-kube-api-access-xk9mg\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.718269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717772 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-podinfo\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.718269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.717808 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-envoy\") pod \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\" (UID: \"ede1f6ef-2a8d-40e5-b6a0-83330a3bc667\") " Sep 10 16:30:21.718927 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.718902 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.719581 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.719490 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:21.719853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.719830 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:21.720199 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.720139 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:21.721828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.721567 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:21.723530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.723490 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-token" (OuterVolumeSpecName: "istio-token") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:21.725619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.725556 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:30:21.725778 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.725757 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-kube-api-access-xk9mg" (OuterVolumeSpecName: "kube-api-access-xk9mg") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "kube-api-access-xk9mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:21.727489 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.727440 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" (UID: "ede1f6ef-2a8d-40e5-b6a0-83330a3bc667"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:21.820490 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.820458 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.820490 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.820490 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.820962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.820506 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.820962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.820518 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.820962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.820532 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.820962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.820545 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.820962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.820556 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.820962 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.820571 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk9mg\" (UniqueName: \"kubernetes.io/projected/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667-kube-api-access-xk9mg\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:21.938872 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.938834 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8"] Sep 10 16:30:21.940171 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:30:21.940132 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bebef4_0b4d_4792_9851_920f01946466.slice/crio-22b8d39dcf43dcf4f32b07bfd93c612d3e03d502cebbd2f3b22d3208e3973b18 WatchSource:0}: Error finding container 22b8d39dcf43dcf4f32b07bfd93c612d3e03d502cebbd2f3b22d3208e3973b18: Status 404 returned error can't find the container with id 22b8d39dcf43dcf4f32b07bfd93c612d3e03d502cebbd2f3b22d3208e3973b18 Sep 10 16:30:21.987118 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.987089 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g"] Sep 10 16:30:21.988982 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:30:21.988953 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad1f679_22f9_4b64_9a42_89e1e0cc3c63.slice/crio-080408883bc547284aa74cc6c32518300b9e74d6159968f2e842390f5f42f41d WatchSource:0}: Error finding container 080408883bc547284aa74cc6c32518300b9e74d6159968f2e842390f5f42f41d: Status 404 returned error can't find the container with id 080408883bc547284aa74cc6c32518300b9e74d6159968f2e842390f5f42f41d Sep 10 16:30:21.994429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:21.994402 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z"] Sep 10 16:30:22.004435 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.004406 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-25d77f0b-openshift-default-76754785cd-6pb9z"] Sep 10 16:30:22.593209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.593124 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:22.677202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.677161 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" event={"ID":"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63","Type":"ContainerStarted","Data":"95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057"} Sep 10 16:30:22.677371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.677263 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" event={"ID":"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63","Type":"ContainerStarted","Data":"080408883bc547284aa74cc6c32518300b9e74d6159968f2e842390f5f42f41d"} Sep 10 16:30:22.677371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.677295 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:22.679507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.679475 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" event={"ID":"e7bebef4-0b4d-4792-9851-920f01946466","Type":"ContainerStarted","Data":"9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3"} Sep 10 16:30:22.679637 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.679569 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:22.679637 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.679605 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" event={"ID":"e7bebef4-0b4d-4792-9851-920f01946466","Type":"ContainerStarted","Data":"22b8d39dcf43dcf4f32b07bfd93c612d3e03d502cebbd2f3b22d3208e3973b18"} Sep 10 16:30:22.698997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.698936 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" podStartSLOduration=1.698918495 podStartE2EDuration="1.698918495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:22.696603448 +0000 UTC m=+773.422922398" watchObservedRunningTime="2026-09-10 16:30:22.698918495 +0000 UTC m=+773.425237425" Sep 10 16:30:22.717616 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:22.717559 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" podStartSLOduration=1.7175432800000001 podStartE2EDuration="1.71754328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:22.716168908 +0000 UTC m=+773.442487852" watchObservedRunningTime="2026-09-10 16:30:22.71754328 +0000 UTC m=+773.443862231" Sep 10 16:30:23.286294 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:23.286259 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:23.484147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:23.484112 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:30:23.758377 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:23.758340 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" path="/var/lib/kubelet/pods/ede1f6ef-2a8d-40e5-b6a0-83330a3bc667/volumes" Sep 10 16:30:24.179507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:24.179415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:33.687598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:33.687566 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:33.688063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:33.687649 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:39.072529 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:39.072490 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:30:41.800134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:41.800101 2562 generic.go:356] "Generic (PLEG): container finished" podID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerID="03612f8233ef12a8fa4c12de6fcbce9ba5aaab7b7ce0e8b4ce1340479c4a2993" exitCode=0 Sep 10 16:30:41.800559 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:41.800184 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" event={"ID":"1f1eedca-daae-4f5d-87a6-55997c3dc378","Type":"ContainerDied","Data":"03612f8233ef12a8fa4c12de6fcbce9ba5aaab7b7ce0e8b4ce1340479c4a2993"} Sep 10 16:30:41.800745 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:41.800728 2562 scope.go:117] "RemoveContainer" containerID="03612f8233ef12a8fa4c12de6fcbce9ba5aaab7b7ce0e8b4ce1340479c4a2993" Sep 10 16:30:41.801950 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:41.801919 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:41.802098 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:41.802002 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:41.802098 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:41.802068 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:42.808031 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:42.807994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" event={"ID":"1f1eedca-daae-4f5d-87a6-55997c3dc378","Type":"ContainerStarted","Data":"22295bb1812ab682abb00f80c9aa43622684a505f4f297e731cc286d540085c0"} Sep 10 16:30:42.944350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:42.944312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:43.543881 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:43.543846 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:43.544601 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:43.544575 2562 patch_prober.go:28] interesting pod/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 16:30:43.544799 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:43.544771 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 16:30:43.867438 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:43.866848 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:44.544001 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:44.543958 2562 patch_prober.go:28] interesting pod/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 16:30:44.544223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:44.544057 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 16:30:44.580322 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:44.580289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:45.137302 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.137264 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh"] Sep 10 16:30:45.137770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.137753 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" Sep 10 16:30:45.137770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.137770 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" Sep 10 16:30:45.137910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.137783 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" Sep 10 16:30:45.137910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.137789 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" Sep 10 16:30:45.137910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.137885 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" Sep 10 16:30:45.137910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.137901 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede1f6ef-2a8d-40e5-b6a0-83330a3bc667" containerName="istio-proxy" Sep 10 16:30:45.141854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.141826 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" Sep 10 16:30:45.145324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.145293 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:30:45.145651 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.145627 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:30:45.150562 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.150529 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh"] Sep 10 16:30:45.281909 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.281861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caca2e3b-62a0-4567-ae11-6c207445595a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh\" (UID: \"caca2e3b-62a0-4567-ae11-6c207445595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" Sep 10 16:30:45.383147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.383107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caca2e3b-62a0-4567-ae11-6c207445595a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh\" (UID: \"caca2e3b-62a0-4567-ae11-6c207445595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" Sep 10 16:30:45.383767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.383744 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caca2e3b-62a0-4567-ae11-6c207445595a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh\" (UID: \"caca2e3b-62a0-4567-ae11-6c207445595a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" Sep 10 16:30:45.435845 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.435759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:45.473783 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.473743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" Sep 10 16:30:45.545661 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.545621 2562 patch_prober.go:28] interesting pod/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 16:30:45.563375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.563267 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 16:30:45.763692 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.763654 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh"] Sep 10 16:30:45.773229 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:30:45.773192 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaca2e3b_62a0_4567_ae11_6c207445595a.slice/crio-6c60ff1c62d070135b8d907d75118a6fd0eba616634e3753fdfc4d6745830188 WatchSource:0}: Error finding container 6c60ff1c62d070135b8d907d75118a6fd0eba616634e3753fdfc4d6745830188: Status 404 returned error can't find the container with id 6c60ff1c62d070135b8d907d75118a6fd0eba616634e3753fdfc4d6745830188 Sep 10 16:30:45.824196 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:45.824164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" event={"ID":"caca2e3b-62a0-4567-ae11-6c207445595a","Type":"ContainerStarted","Data":"6c60ff1c62d070135b8d907d75118a6fd0eba616634e3753fdfc4d6745830188"} Sep 10 16:30:46.401559 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:46.401467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:46.544226 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:46.544182 2562 patch_prober.go:28] interesting pod/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 16:30:46.544438 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:46.544254 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 16:30:46.830694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:46.830659 2562 generic.go:356] "Generic (PLEG): container finished" podID="caca2e3b-62a0-4567-ae11-6c207445595a" containerID="b1d58f487d49b04d4d46ad7ef1b31507f0924b41fee6dc4d16a169c358185837" exitCode=0 Sep 10 16:30:46.830899 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:46.830743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" event={"ID":"caca2e3b-62a0-4567-ae11-6c207445595a","Type":"ContainerDied","Data":"b1d58f487d49b04d4d46ad7ef1b31507f0924b41fee6dc4d16a169c358185837"} Sep 10 16:30:47.220386 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:47.220299 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:47.543556 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:47.543508 2562 patch_prober.go:28] interesting pod/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 16:30:47.544092 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:47.543581 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 16:30:47.991809 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:47.991781 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" Sep 10 16:30:48.089608 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.089516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:48.113602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.113569 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caca2e3b-62a0-4567-ae11-6c207445595a-maas-api-service-ca\") pod \"caca2e3b-62a0-4567-ae11-6c207445595a\" (UID: \"caca2e3b-62a0-4567-ae11-6c207445595a\") " Sep 10 16:30:48.113911 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.113889 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caca2e3b-62a0-4567-ae11-6c207445595a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "caca2e3b-62a0-4567-ae11-6c207445595a" (UID: "caca2e3b-62a0-4567-ae11-6c207445595a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:48.215286 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.215244 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caca2e3b-62a0-4567-ae11-6c207445595a-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:48.548171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.548137 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:48.548552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.548418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:48.549246 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.549229 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:48.778251 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.778210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:48.845891 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.845810 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" Sep 10 16:30:48.845891 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.845851 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh" event={"ID":"caca2e3b-62a0-4567-ae11-6c207445595a","Type":"ContainerDied","Data":"6c60ff1c62d070135b8d907d75118a6fd0eba616634e3753fdfc4d6745830188"} Sep 10 16:30:48.846111 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:48.845896 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c60ff1c62d070135b8d907d75118a6fd0eba616634e3753fdfc4d6745830188" Sep 10 16:30:49.578670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:49.578638 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:50.761458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:50.761415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:51.346563 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.346531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:51.369383 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.369347 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g"] Sep 10 16:30:51.369607 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.369583 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" podUID="4ad1f679-22f9-4b64-9a42-89e1e0cc3c63" containerName="payload-processing" containerID="cri-o://95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057" gracePeriod=30 Sep 10 16:30:51.383267 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.383232 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8"] Sep 10 16:30:51.383594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.383503 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" podUID="e7bebef4-0b4d-4792-9851-920f01946466" containerName="payload-pre-processing" containerID="cri-o://9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3" gracePeriod=30 Sep 10 16:30:51.418915 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.418884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:51.511757 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.511703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:51.532784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.532745 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh"] Sep 10 16:30:51.535364 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.535335 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2238d044-nwbrh"] Sep 10 16:30:51.625114 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.620388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:51.671260 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.671100 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6"] Sep 10 16:30:51.671419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.671314 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" containerID="cri-o://22295bb1812ab682abb00f80c9aa43622684a505f4f297e731cc286d540085c0" gracePeriod=30 Sep 10 16:30:51.682235 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.682201 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:51.692400 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.692367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:51.721462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.721436 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:51.767908 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.767815 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs49n\" (UniqueName: \"kubernetes.io/projected/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-kube-api-access-vs49n\") pod \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\" (UID: \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\") " Sep 10 16:30:51.769917 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.769167 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-plugins-config-volume\") pod \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\" (UID: \"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63\") " Sep 10 16:30:51.771997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.771956 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ad1f679-22f9-4b64-9a42-89e1e0cc3c63" (UID: "4ad1f679-22f9-4b64-9a42-89e1e0cc3c63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:51.774535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.774490 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caca2e3b-62a0-4567-ae11-6c207445595a" path="/var/lib/kubelet/pods/caca2e3b-62a0-4567-ae11-6c207445595a/volumes" Sep 10 16:30:51.789489 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.789412 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-kube-api-access-vs49n" (OuterVolumeSpecName: "kube-api-access-vs49n") pod "4ad1f679-22f9-4b64-9a42-89e1e0cc3c63" (UID: "4ad1f679-22f9-4b64-9a42-89e1e0cc3c63"). InnerVolumeSpecName "kube-api-access-vs49n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:51.868647 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.868609 2562 generic.go:356] "Generic (PLEG): container finished" podID="e7bebef4-0b4d-4792-9851-920f01946466" containerID="9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3" exitCode=0 Sep 10 16:30:51.868828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.868652 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" Sep 10 16:30:51.868828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.868692 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" event={"ID":"e7bebef4-0b4d-4792-9851-920f01946466","Type":"ContainerDied","Data":"9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3"} Sep 10 16:30:51.868828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.868742 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8" event={"ID":"e7bebef4-0b4d-4792-9851-920f01946466","Type":"ContainerDied","Data":"22b8d39dcf43dcf4f32b07bfd93c612d3e03d502cebbd2f3b22d3208e3973b18"} Sep 10 16:30:51.868828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.868766 2562 scope.go:117] "RemoveContainer" containerID="9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3" Sep 10 16:30:51.870048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.870014 2562 generic.go:356] "Generic (PLEG): container finished" podID="4ad1f679-22f9-4b64-9a42-89e1e0cc3c63" containerID="95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057" exitCode=0 Sep 10 16:30:51.870127 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.870080 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" Sep 10 16:30:51.870127 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.870091 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" event={"ID":"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63","Type":"ContainerDied","Data":"95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057"} Sep 10 16:30:51.870236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.870137 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g" event={"ID":"4ad1f679-22f9-4b64-9a42-89e1e0cc3c63","Type":"ContainerDied","Data":"080408883bc547284aa74cc6c32518300b9e74d6159968f2e842390f5f42f41d"} Sep 10 16:30:51.871697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.871660 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bebef4-0b4d-4792-9851-920f01946466-plugins-config-volume\") pod \"e7bebef4-0b4d-4792-9851-920f01946466\" (UID: \"e7bebef4-0b4d-4792-9851-920f01946466\") " Sep 10 16:30:51.871835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.871768 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzcj7\" (UniqueName: \"kubernetes.io/projected/e7bebef4-0b4d-4792-9851-920f01946466-kube-api-access-nzcj7\") pod \"e7bebef4-0b4d-4792-9851-920f01946466\" (UID: \"e7bebef4-0b4d-4792-9851-920f01946466\") " Sep 10 16:30:51.872088 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.872060 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7bebef4-0b4d-4792-9851-920f01946466-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7bebef4-0b4d-4792-9851-920f01946466" (UID: "e7bebef4-0b4d-4792-9851-920f01946466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:51.872452 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.872286 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bebef4-0b4d-4792-9851-920f01946466-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:51.872452 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.872402 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:51.872452 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.872421 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs49n\" (UniqueName: \"kubernetes.io/projected/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63-kube-api-access-vs49n\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:51.874419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.874395 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bebef4-0b4d-4792-9851-920f01946466-kube-api-access-nzcj7" (OuterVolumeSpecName: "kube-api-access-nzcj7") pod "e7bebef4-0b4d-4792-9851-920f01946466" (UID: "e7bebef4-0b4d-4792-9851-920f01946466"). InnerVolumeSpecName "kube-api-access-nzcj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:51.878102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.878080 2562 scope.go:117] "RemoveContainer" containerID="9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3" Sep 10 16:30:51.878512 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:30:51.878394 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3\": container with ID starting with 9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3 not found: ID does not exist" containerID="9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3" Sep 10 16:30:51.878512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.878422 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3"} err="failed to get container status \"9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3\": rpc error: code = NotFound desc = could not find container \"9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3\": container with ID starting with 9f44c7dbebdfee3de7b3240e7c5c6f3c6b57767b9e5b1f40b8ebd99e760e45a3 not found: ID does not exist" Sep 10 16:30:51.878512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.878437 2562 scope.go:117] "RemoveContainer" containerID="95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057" Sep 10 16:30:51.887010 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.886992 2562 scope.go:117] "RemoveContainer" containerID="95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057" Sep 10 16:30:51.887418 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:30:51.887400 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057\": container with ID starting with 95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057 not found: ID does not exist" containerID="95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057" Sep 10 16:30:51.887495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.887424 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057"} err="failed to get container status \"95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057\": rpc error: code = NotFound desc = could not find container \"95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057\": container with ID starting with 95c8f02779095d5e876f61decad64232ddb30775f2a05b6f3762f212994a7057 not found: ID does not exist" Sep 10 16:30:51.926461 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.926416 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g"] Sep 10 16:30:51.935335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.935299 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2238d044-bf6f5b48b-kqt8g"] Sep 10 16:30:51.973634 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:51.973594 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzcj7\" (UniqueName: \"kubernetes.io/projected/e7bebef4-0b4d-4792-9851-920f01946466-kube-api-access-nzcj7\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:52.040300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.040257 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:52.140664 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.140576 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:52.195158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.195126 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8"] Sep 10 16:30:52.198372 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.198345 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2238d044-594bb87578-rjqt8"] Sep 10 16:30:52.398134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398027 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8"] Sep 10 16:30:52.398545 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398528 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caca2e3b-62a0-4567-ae11-6c207445595a" containerName="revoke-keys" Sep 10 16:30:52.398545 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398545 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="caca2e3b-62a0-4567-ae11-6c207445595a" containerName="revoke-keys" Sep 10 16:30:52.398646 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398560 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad1f679-22f9-4b64-9a42-89e1e0cc3c63" containerName="payload-processing" Sep 10 16:30:52.398646 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398566 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad1f679-22f9-4b64-9a42-89e1e0cc3c63" containerName="payload-processing" Sep 10 16:30:52.398646 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398581 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7bebef4-0b4d-4792-9851-920f01946466" containerName="payload-pre-processing" Sep 10 16:30:52.398646 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398586 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7bebef4-0b4d-4792-9851-920f01946466" containerName="payload-pre-processing" Sep 10 16:30:52.398795 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398665 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7bebef4-0b4d-4792-9851-920f01946466" containerName="payload-pre-processing" Sep 10 16:30:52.398795 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398676 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="caca2e3b-62a0-4567-ae11-6c207445595a" containerName="revoke-keys" Sep 10 16:30:52.398795 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.398684 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad1f679-22f9-4b64-9a42-89e1e0cc3c63" containerName="payload-processing" Sep 10 16:30:52.401917 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.401897 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.405089 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.405068 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f89c54f2-openshift-default-dockercfg-v88gx\"" Sep 10 16:30:52.412694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.412664 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8"] Sep 10 16:30:52.421700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.421671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:52.479349 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-certs\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.479512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-envoy\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.479512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-token\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.479512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7k7\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-kube-api-access-2f7k7\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.479512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-data\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.479654 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479574 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-credential-socket\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.479654 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-socket\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.479654 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c936b82f-e678-482f-ac92-cf4d91fb50ce-istiod-ca-cert\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.479788 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.479683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-podinfo\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.529093 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.529028 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-data\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-credential-socket\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-socket\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c936b82f-e678-482f-ac92-cf4d91fb50ce-istiod-ca-cert\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-podinfo\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-certs\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-envoy\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-token\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.580935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7k7\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-kube-api-access-2f7k7\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.581807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-data\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.582116 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-credential-socket\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.582400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-socket\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.583443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-certs\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.584676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.583456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c936b82f-e678-482f-ac92-cf4d91fb50ce-istiod-ca-cert\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.586569 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.586509 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-podinfo\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.586983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.586952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-envoy\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.592720 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.592667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7k7\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-kube-api-access-2f7k7\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.592851 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.592778 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-token\") pod \"e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:52.714985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:52.714891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:53.001884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.001853 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8"] Sep 10 16:30:53.002996 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.002969 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:53.009252 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.009218 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:53.009395 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.009287 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:53.009395 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.009317 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:30:53.686665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.686630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:53.758054 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.758008 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad1f679-22f9-4b64-9a42-89e1e0cc3c63" path="/var/lib/kubelet/pods/4ad1f679-22f9-4b64-9a42-89e1e0cc3c63/volumes" Sep 10 16:30:53.758443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.758428 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7bebef4-0b4d-4792-9851-920f01946466" path="/var/lib/kubelet/pods/e7bebef4-0b4d-4792-9851-920f01946466/volumes" Sep 10 16:30:53.883896 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.883857 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" event={"ID":"c936b82f-e678-482f-ac92-cf4d91fb50ce","Type":"ContainerStarted","Data":"22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a"} Sep 10 16:30:53.884084 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.883921 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" event={"ID":"c936b82f-e678-482f-ac92-cf4d91fb50ce","Type":"ContainerStarted","Data":"3bb13779c5f25389af70dd034a1fdff0384dfd008503f4e0e274a74df055f82d"} Sep 10 16:30:53.913697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:53.913642 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" podStartSLOduration=1.9136244900000001 podStartE2EDuration="1.91362449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:53.912323706 +0000 UTC m=+804.638642650" watchObservedRunningTime="2026-09-10 16:30:53.91362449 +0000 UTC m=+804.639943416" Sep 10 16:30:54.546955 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:54.546917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:54.715580 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:54.715546 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:54.720856 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:54.720828 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:54.889528 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:54.889446 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:54.890550 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:54.890532 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:30:55.261424 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:55.261386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:55.358455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:55.358420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:55.493561 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:55.493528 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:56.249020 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:56.248986 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:30:56.333445 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:56.333411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:56.903826 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:56.903560 2562 generic.go:356] "Generic (PLEG): container finished" podID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerID="22295bb1812ab682abb00f80c9aa43622684a505f4f297e731cc286d540085c0" exitCode=0 Sep 10 16:30:56.903826 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:56.903820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" event={"ID":"1f1eedca-daae-4f5d-87a6-55997c3dc378","Type":"ContainerDied","Data":"22295bb1812ab682abb00f80c9aa43622684a505f4f297e731cc286d540085c0"} Sep 10 16:30:56.904135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:56.903881 2562 scope.go:117] "RemoveContainer" containerID="03612f8233ef12a8fa4c12de6fcbce9ba5aaab7b7ce0e8b4ce1340479c4a2993" Sep 10 16:30:57.029297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.029273 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:57.128422 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128319 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-data\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.128422 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128403 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1eedca-daae-4f5d-87a6-55997c3dc378-istiod-ca-cert\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.128422 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128421 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-socket\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.128719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128451 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-token\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.128719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128487 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-envoy\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.128719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128540 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjssl\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-kube-api-access-sjssl\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.128719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128566 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-credential-socket\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.128719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128606 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-certs\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.128719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128645 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-podinfo\") pod \"1f1eedca-daae-4f5d-87a6-55997c3dc378\" (UID: \"1f1eedca-daae-4f5d-87a6-55997c3dc378\") " Sep 10 16:30:57.129072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128637 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:57.129072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128708 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:57.129072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128888 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1eedca-daae-4f5d-87a6-55997c3dc378-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:30:57.129072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.128890 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:57.129274 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.129227 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:57.130916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.130890 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:30:57.131058 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.130980 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:30:57.131058 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.131005 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:57.131340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.131323 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-kube-api-access-sjssl" (OuterVolumeSpecName: "kube-api-access-sjssl") pod "1f1eedca-daae-4f5d-87a6-55997c3dc378" (UID: "1f1eedca-daae-4f5d-87a6-55997c3dc378"). InnerVolumeSpecName "kube-api-access-sjssl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:30:57.202507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.202460 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:57.230527 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230490 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1eedca-daae-4f5d-87a6-55997c3dc378-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.230527 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230520 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.230527 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230530 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.230763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230541 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.230763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230554 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjssl\" (UniqueName: \"kubernetes.io/projected/1f1eedca-daae-4f5d-87a6-55997c3dc378-kube-api-access-sjssl\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.230763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230568 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.230763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230577 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.230763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230588 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.230763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.230596 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1eedca-daae-4f5d-87a6-55997c3dc378-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:30:57.910874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.910834 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" event={"ID":"1f1eedca-daae-4f5d-87a6-55997c3dc378","Type":"ContainerDied","Data":"8abedf353d61a179d567f43877045e1c3a72f4d6362da6d278db74c4e3c14841"} Sep 10 16:30:57.910874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.910857 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6" Sep 10 16:30:57.911415 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.910903 2562 scope.go:117] "RemoveContainer" containerID="22295bb1812ab682abb00f80c9aa43622684a505f4f297e731cc286d540085c0" Sep 10 16:30:57.938385 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.938355 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6"] Sep 10 16:30:57.942905 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.942876 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2238d044-gw-openshift-default-65c8f7b99b-fzxl6"] Sep 10 16:30:57.972622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:57.972582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:58.644954 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:58.644914 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:58.719056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:58.719000 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:58.987673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:58.987594 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:59.117801 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.117764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:59.216798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.216765 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:30:59.228033 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.228002 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6"] Sep 10 16:30:59.228460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.228445 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" Sep 10 16:30:59.228516 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.228461 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" Sep 10 16:30:59.228516 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.228475 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" Sep 10 16:30:59.228516 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.228481 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" Sep 10 16:30:59.228610 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.228539 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" Sep 10 16:30:59.231400 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.231382 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.234513 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.234488 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f89c54f2-dockercfg-6h7d9\"" Sep 10 16:30:59.234642 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.234488 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f89c54f2\"" Sep 10 16:30:59.244716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.244690 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6"] Sep 10 16:30:59.266294 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.266263 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd"] Sep 10 16:30:59.266734 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.266721 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" containerName="istio-proxy" Sep 10 16:30:59.268649 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.268632 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.281574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.281545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd"] Sep 10 16:30:59.352005 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.351967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90794fbd-6817-459d-befc-1676e26cd169-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6\" (UID: \"90794fbd-6817-459d-befc-1676e26cd169\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.352179 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.352117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77t45\" (UniqueName: \"kubernetes.io/projected/90794fbd-6817-459d-befc-1676e26cd169-kube-api-access-77t45\") pod \"payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6\" (UID: \"90794fbd-6817-459d-befc-1676e26cd169\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.453146 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.453102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90794fbd-6817-459d-befc-1676e26cd169-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6\" (UID: \"90794fbd-6817-459d-befc-1676e26cd169\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.453329 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.453175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz4nq\" (UniqueName: \"kubernetes.io/projected/cc90796e-b637-43c2-8005-433f09f00300-kube-api-access-pz4nq\") pod \"payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd\" (UID: \"cc90796e-b637-43c2-8005-433f09f00300\") " pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.453329 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.453239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc90796e-b637-43c2-8005-433f09f00300-plugins-config-volume\") pod \"payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd\" (UID: \"cc90796e-b637-43c2-8005-433f09f00300\") " pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.453329 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.453277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77t45\" (UniqueName: \"kubernetes.io/projected/90794fbd-6817-459d-befc-1676e26cd169-kube-api-access-77t45\") pod \"payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6\" (UID: \"90794fbd-6817-459d-befc-1676e26cd169\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.453801 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.453778 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90794fbd-6817-459d-befc-1676e26cd169-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6\" (UID: \"90794fbd-6817-459d-befc-1676e26cd169\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.462026 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.461997 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77t45\" (UniqueName: \"kubernetes.io/projected/90794fbd-6817-459d-befc-1676e26cd169-kube-api-access-77t45\") pod \"payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6\" (UID: \"90794fbd-6817-459d-befc-1676e26cd169\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.542164 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.542122 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.554554 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.554524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc90796e-b637-43c2-8005-433f09f00300-plugins-config-volume\") pod \"payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd\" (UID: \"cc90796e-b637-43c2-8005-433f09f00300\") " pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.554676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.554651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz4nq\" (UniqueName: \"kubernetes.io/projected/cc90796e-b637-43c2-8005-433f09f00300-kube-api-access-pz4nq\") pod \"payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd\" (UID: \"cc90796e-b637-43c2-8005-433f09f00300\") " pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.555469 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.555442 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc90796e-b637-43c2-8005-433f09f00300-plugins-config-volume\") pod \"payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd\" (UID: \"cc90796e-b637-43c2-8005-433f09f00300\") " pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.566548 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.566518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz4nq\" (UniqueName: \"kubernetes.io/projected/cc90796e-b637-43c2-8005-433f09f00300-kube-api-access-pz4nq\") pod \"payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd\" (UID: \"cc90796e-b637-43c2-8005-433f09f00300\") " pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.579343 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.579307 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.757102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.757067 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f1eedca-daae-4f5d-87a6-55997c3dc378" path="/var/lib/kubelet/pods/1f1eedca-daae-4f5d-87a6-55997c3dc378/volumes" Sep 10 16:30:59.817840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.817815 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6"] Sep 10 16:30:59.819641 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:30:59.819613 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90794fbd_6817_459d_befc_1676e26cd169.slice/crio-dd48a80b4e5c34c5d34738feaa9e1ff06a0ed896765ab20a720c66a0391f6d0d WatchSource:0}: Error finding container dd48a80b4e5c34c5d34738feaa9e1ff06a0ed896765ab20a720c66a0391f6d0d: Status 404 returned error can't find the container with id dd48a80b4e5c34c5d34738feaa9e1ff06a0ed896765ab20a720c66a0391f6d0d Sep 10 16:30:59.859706 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.859681 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd"] Sep 10 16:30:59.862070 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:30:59.862030 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc90796e_b637_43c2_8005_433f09f00300.slice/crio-44d44da62fd20fc702423b293c7b48ab6f6678e13d193d54b5569f1ee9fa805b WatchSource:0}: Error finding container 44d44da62fd20fc702423b293c7b48ab6f6678e13d193d54b5569f1ee9fa805b: Status 404 returned error can't find the container with id 44d44da62fd20fc702423b293c7b48ab6f6678e13d193d54b5569f1ee9fa805b Sep 10 16:30:59.924904 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.924863 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" event={"ID":"90794fbd-6817-459d-befc-1676e26cd169","Type":"ContainerStarted","Data":"c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7"} Sep 10 16:30:59.925105 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.925025 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:30:59.925105 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.925074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" event={"ID":"90794fbd-6817-459d-befc-1676e26cd169","Type":"ContainerStarted","Data":"dd48a80b4e5c34c5d34738feaa9e1ff06a0ed896765ab20a720c66a0391f6d0d"} Sep 10 16:30:59.926673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.926644 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" event={"ID":"cc90796e-b637-43c2-8005-433f09f00300","Type":"ContainerStarted","Data":"0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631"} Sep 10 16:30:59.926778 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.926692 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:30:59.926778 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.926708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" event={"ID":"cc90796e-b637-43c2-8005-433f09f00300","Type":"ContainerStarted","Data":"44d44da62fd20fc702423b293c7b48ab6f6678e13d193d54b5569f1ee9fa805b"} Sep 10 16:30:59.941517 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.941456 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" podStartSLOduration=0.941435362 podStartE2EDuration="941.435362ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:59.941023057 +0000 UTC m=+810.667342016" watchObservedRunningTime="2026-09-10 16:30:59.941435362 +0000 UTC m=+810.667754305" Sep 10 16:30:59.958240 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:30:59.958186 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" podStartSLOduration=0.95816892 podStartE2EDuration="958.16892ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:30:59.957522208 +0000 UTC m=+810.683841138" watchObservedRunningTime="2026-09-10 16:30:59.95816892 +0000 UTC m=+810.684487852" Sep 10 16:31:07.150278 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:07.150238 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:10.937914 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:10.937884 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:31:10.938479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:10.937958 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:31:12.951507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:12.951472 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:31:13.675739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:13.675689 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:15.024156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:15.024120 2562 generic.go:356] "Generic (PLEG): container finished" podID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerID="22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a" exitCode=0 Sep 10 16:31:15.024693 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:15.024192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" event={"ID":"c936b82f-e678-482f-ac92-cf4d91fb50ce","Type":"ContainerDied","Data":"22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a"} Sep 10 16:31:15.024771 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:15.024759 2562 scope.go:117] "RemoveContainer" containerID="22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a" Sep 10 16:31:15.025880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:15.025853 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:15.025939 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:15.025923 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:15.025978 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:15.025953 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:16.031269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:16.031228 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" event={"ID":"c936b82f-e678-482f-ac92-cf4d91fb50ce","Type":"ContainerStarted","Data":"abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d"} Sep 10 16:31:16.715671 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:16.715635 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:31:16.715890 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:16.715868 2562 patch_prober.go:28] interesting pod/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 16:31:16.715939 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:16.715919 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 16:31:17.715522 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:17.715479 2562 patch_prober.go:28] interesting pod/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 16:31:17.715911 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:17.715554 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 16:31:18.716287 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:18.716247 2562 patch_prober.go:28] interesting pod/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 16:31:18.716688 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:18.716310 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 16:31:19.716024 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:19.715985 2562 patch_prober.go:28] interesting pod/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 16:31:19.716213 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:19.716113 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 16:31:20.715768 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:20.715727 2562 patch_prober.go:28] interesting pod/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 16:31:20.716231 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:20.715800 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 16:31:20.754793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:20.754749 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:21.403605 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:21.403542 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:21.716030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:21.715947 2562 patch_prober.go:28] interesting pod/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 16:31:21.716030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:21.716001 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 16:31:22.142570 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:22.142535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:22.720089 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:22.720031 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:31:22.720496 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:22.720474 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:31:22.721102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:22.721081 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:31:22.921549 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:22.921500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:23.854089 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:23.854032 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:26.052051 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.051997 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw"] Sep 10 16:31:26.054952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.054933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" Sep 10 16:31:26.057900 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.057880 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:31:26.057998 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.057942 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:31:26.064487 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.064463 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw"] Sep 10 16:31:26.155235 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.155196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31210ad-cf41-42a3-bc59-e3a0fba7c04c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw\" (UID: \"c31210ad-cf41-42a3-bc59-e3a0fba7c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" Sep 10 16:31:26.256307 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.256259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31210ad-cf41-42a3-bc59-e3a0fba7c04c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw\" (UID: \"c31210ad-cf41-42a3-bc59-e3a0fba7c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" Sep 10 16:31:26.256900 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.256877 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31210ad-cf41-42a3-bc59-e3a0fba7c04c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw\" (UID: \"c31210ad-cf41-42a3-bc59-e3a0fba7c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" Sep 10 16:31:26.366872 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.366774 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" Sep 10 16:31:26.639214 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.639160 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw"] Sep 10 16:31:26.641736 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:31:26.641715 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31210ad_cf41_42a3_bc59_e3a0fba7c04c.slice/crio-3f431625ce775451d214e94c34c9f38b7243c70210a808116a32fff4e713e8b0 WatchSource:0}: Error finding container 3f431625ce775451d214e94c34c9f38b7243c70210a808116a32fff4e713e8b0: Status 404 returned error can't find the container with id 3f431625ce775451d214e94c34c9f38b7243c70210a808116a32fff4e713e8b0 Sep 10 16:31:26.857081 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:26.857045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:27.102211 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:27.102176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" event={"ID":"c31210ad-cf41-42a3-bc59-e3a0fba7c04c","Type":"ContainerStarted","Data":"b15db04fa21cf6694dc1c0faacbdf8da3abab4ca0db3277cf775aebc05bd374f"} Sep 10 16:31:27.102650 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:27.102234 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" event={"ID":"c31210ad-cf41-42a3-bc59-e3a0fba7c04c","Type":"ContainerStarted","Data":"3f431625ce775451d214e94c34c9f38b7243c70210a808116a32fff4e713e8b0"} Sep 10 16:31:27.124839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:27.124782 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" podStartSLOduration=1.124765342 podStartE2EDuration="1.124765342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:27.123124082 +0000 UTC m=+837.849443036" watchObservedRunningTime="2026-09-10 16:31:27.124765342 +0000 UTC m=+837.851084273" Sep 10 16:31:27.681093 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:27.680996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:28.109909 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:28.109874 2562 generic.go:356] "Generic (PLEG): container finished" podID="c31210ad-cf41-42a3-bc59-e3a0fba7c04c" containerID="b15db04fa21cf6694dc1c0faacbdf8da3abab4ca0db3277cf775aebc05bd374f" exitCode=0 Sep 10 16:31:28.110331 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:28.109962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" event={"ID":"c31210ad-cf41-42a3-bc59-e3a0fba7c04c","Type":"ContainerDied","Data":"b15db04fa21cf6694dc1c0faacbdf8da3abab4ca0db3277cf775aebc05bd374f"} Sep 10 16:31:28.411544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:28.411453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:29.241209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:29.241178 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:29.250506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:29.250482 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" Sep 10 16:31:29.388854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:29.388758 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31210ad-cf41-42a3-bc59-e3a0fba7c04c-maas-api-service-ca\") pod \"c31210ad-cf41-42a3-bc59-e3a0fba7c04c\" (UID: \"c31210ad-cf41-42a3-bc59-e3a0fba7c04c\") " Sep 10 16:31:29.389213 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:29.389183 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31210ad-cf41-42a3-bc59-e3a0fba7c04c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c31210ad-cf41-42a3-bc59-e3a0fba7c04c" (UID: "c31210ad-cf41-42a3-bc59-e3a0fba7c04c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:31:29.490375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:29.490338 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31210ad-cf41-42a3-bc59-e3a0fba7c04c-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:29.902966 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:29.902928 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:30.122391 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:30.122291 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" Sep 10 16:31:30.122391 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:30.122322 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw" event={"ID":"c31210ad-cf41-42a3-bc59-e3a0fba7c04c","Type":"ContainerDied","Data":"3f431625ce775451d214e94c34c9f38b7243c70210a808116a32fff4e713e8b0"} Sep 10 16:31:30.122391 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:30.122376 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f431625ce775451d214e94c34c9f38b7243c70210a808116a32fff4e713e8b0" Sep 10 16:31:30.156530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:30.156485 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:31:33.921283 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:33.921246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:36.047108 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:36.047075 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:36.109176 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:36.109142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:36.309876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:36.309791 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:37.042747 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.042704 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:37.078125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.078092 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd"] Sep 10 16:31:37.078583 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.078320 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" podUID="cc90796e-b637-43c2-8005-433f09f00300" containerName="payload-processing" containerID="cri-o://0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631" gracePeriod=30 Sep 10 16:31:37.087462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.087431 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6"] Sep 10 16:31:37.087725 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.087689 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" podUID="90794fbd-6817-459d-befc-1676e26cd169" containerName="payload-pre-processing" containerID="cri-o://c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7" gracePeriod=30 Sep 10 16:31:37.133054 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.133010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:37.192227 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.192190 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585"] Sep 10 16:31:37.192692 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.192677 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31210ad-cf41-42a3-bc59-e3a0fba7c04c" containerName="revoke-keys" Sep 10 16:31:37.192746 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.192694 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31210ad-cf41-42a3-bc59-e3a0fba7c04c" containerName="revoke-keys" Sep 10 16:31:37.192781 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.192769 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31210ad-cf41-42a3-bc59-e3a0fba7c04c" containerName="revoke-keys" Sep 10 16:31:37.195134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.195112 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" Sep 10 16:31:37.198524 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.198500 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:31:37.198782 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.198763 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:31:37.206734 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.206484 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585"] Sep 10 16:31:37.210272 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.210249 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:37.277197 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.277161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f5ea93-2244-489b-8def-75e68bf3da87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2d818c-8p585\" (UID: \"88f5ea93-2244-489b-8def-75e68bf3da87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" Sep 10 16:31:37.286592 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.286382 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:37.360416 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.360388 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw"] Sep 10 16:31:37.365411 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.365380 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f89c54f2-7ctmw"] Sep 10 16:31:37.372604 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.372576 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:37.376201 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.376182 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:31:37.378539 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.378518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f5ea93-2244-489b-8def-75e68bf3da87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2d818c-8p585\" (UID: \"88f5ea93-2244-489b-8def-75e68bf3da87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" Sep 10 16:31:37.379306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.379286 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f5ea93-2244-489b-8def-75e68bf3da87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2d818c-8p585\" (UID: \"88f5ea93-2244-489b-8def-75e68bf3da87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" Sep 10 16:31:37.407974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.407952 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:31:37.448469 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.448434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:37.479702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.479668 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77t45\" (UniqueName: \"kubernetes.io/projected/90794fbd-6817-459d-befc-1676e26cd169-kube-api-access-77t45\") pod \"90794fbd-6817-459d-befc-1676e26cd169\" (UID: \"90794fbd-6817-459d-befc-1676e26cd169\") " Sep 10 16:31:37.479895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.479744 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc90796e-b637-43c2-8005-433f09f00300-plugins-config-volume\") pod \"cc90796e-b637-43c2-8005-433f09f00300\" (UID: \"cc90796e-b637-43c2-8005-433f09f00300\") " Sep 10 16:31:37.479895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.479763 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz4nq\" (UniqueName: \"kubernetes.io/projected/cc90796e-b637-43c2-8005-433f09f00300-kube-api-access-pz4nq\") pod \"cc90796e-b637-43c2-8005-433f09f00300\" (UID: \"cc90796e-b637-43c2-8005-433f09f00300\") " Sep 10 16:31:37.479895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.479810 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90794fbd-6817-459d-befc-1676e26cd169-plugins-config-volume\") pod \"90794fbd-6817-459d-befc-1676e26cd169\" (UID: \"90794fbd-6817-459d-befc-1676e26cd169\") " Sep 10 16:31:37.480233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.480208 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc90796e-b637-43c2-8005-433f09f00300-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc90796e-b637-43c2-8005-433f09f00300" (UID: "cc90796e-b637-43c2-8005-433f09f00300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:31:37.480313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.480210 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90794fbd-6817-459d-befc-1676e26cd169-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90794fbd-6817-459d-befc-1676e26cd169" (UID: "90794fbd-6817-459d-befc-1676e26cd169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:31:37.481836 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.481815 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90794fbd-6817-459d-befc-1676e26cd169-kube-api-access-77t45" (OuterVolumeSpecName: "kube-api-access-77t45") pod "90794fbd-6817-459d-befc-1676e26cd169" (UID: "90794fbd-6817-459d-befc-1676e26cd169"). InnerVolumeSpecName "kube-api-access-77t45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:31:37.481891 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.481867 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc90796e-b637-43c2-8005-433f09f00300-kube-api-access-pz4nq" (OuterVolumeSpecName: "kube-api-access-pz4nq") pod "cc90796e-b637-43c2-8005-433f09f00300" (UID: "cc90796e-b637-43c2-8005-433f09f00300"). InnerVolumeSpecName "kube-api-access-pz4nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:31:37.512820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.512789 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" Sep 10 16:31:37.581412 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.581380 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc90796e-b637-43c2-8005-433f09f00300-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:37.581412 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.581411 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz4nq\" (UniqueName: \"kubernetes.io/projected/cc90796e-b637-43c2-8005-433f09f00300-kube-api-access-pz4nq\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:37.581606 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.581420 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90794fbd-6817-459d-befc-1676e26cd169-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:37.581606 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.581431 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77t45\" (UniqueName: \"kubernetes.io/projected/90794fbd-6817-459d-befc-1676e26cd169-kube-api-access-77t45\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:37.758599 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.758563 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31210ad-cf41-42a3-bc59-e3a0fba7c04c" path="/var/lib/kubelet/pods/c31210ad-cf41-42a3-bc59-e3a0fba7c04c/volumes" Sep 10 16:31:37.785473 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:37.785444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585"] Sep 10 16:31:37.787758 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:31:37.787732 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f5ea93_2244_489b_8def_75e68bf3da87.slice/crio-eec360634a0ed2c5b475af26a0dc09ad92efbc5932d0ab26cd55cc9af6bbd179 WatchSource:0}: Error finding container eec360634a0ed2c5b475af26a0dc09ad92efbc5932d0ab26cd55cc9af6bbd179: Status 404 returned error can't find the container with id eec360634a0ed2c5b475af26a0dc09ad92efbc5932d0ab26cd55cc9af6bbd179 Sep 10 16:31:38.166762 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.166724 2562 generic.go:356] "Generic (PLEG): container finished" podID="90794fbd-6817-459d-befc-1676e26cd169" containerID="c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7" exitCode=0 Sep 10 16:31:38.167222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.166770 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" Sep 10 16:31:38.167222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.166813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" event={"ID":"90794fbd-6817-459d-befc-1676e26cd169","Type":"ContainerDied","Data":"c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7"} Sep 10 16:31:38.167222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.166861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6" event={"ID":"90794fbd-6817-459d-befc-1676e26cd169","Type":"ContainerDied","Data":"dd48a80b4e5c34c5d34738feaa9e1ff06a0ed896765ab20a720c66a0391f6d0d"} Sep 10 16:31:38.167222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.166883 2562 scope.go:117] "RemoveContainer" containerID="c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7" Sep 10 16:31:38.168829 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.168801 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" event={"ID":"88f5ea93-2244-489b-8def-75e68bf3da87","Type":"ContainerStarted","Data":"270c7f935832721cd2f4d2fccf2c1026d39910a29a9a67c27978786c57c88848"} Sep 10 16:31:38.168968 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.168864 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" event={"ID":"88f5ea93-2244-489b-8def-75e68bf3da87","Type":"ContainerStarted","Data":"eec360634a0ed2c5b475af26a0dc09ad92efbc5932d0ab26cd55cc9af6bbd179"} Sep 10 16:31:38.170372 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.170347 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc90796e-b637-43c2-8005-433f09f00300" containerID="0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631" exitCode=0 Sep 10 16:31:38.170458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.170382 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" Sep 10 16:31:38.170458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.170383 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" event={"ID":"cc90796e-b637-43c2-8005-433f09f00300","Type":"ContainerDied","Data":"0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631"} Sep 10 16:31:38.170540 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.170485 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd" event={"ID":"cc90796e-b637-43c2-8005-433f09f00300","Type":"ContainerDied","Data":"44d44da62fd20fc702423b293c7b48ab6f6678e13d193d54b5569f1ee9fa805b"} Sep 10 16:31:38.179528 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.179505 2562 scope.go:117] "RemoveContainer" containerID="c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7" Sep 10 16:31:38.179846 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:31:38.179827 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7\": container with ID starting with c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7 not found: ID does not exist" containerID="c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7" Sep 10 16:31:38.179923 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.179854 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7"} err="failed to get container status \"c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7\": rpc error: code = NotFound desc = could not find container \"c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7\": container with ID starting with c3f7e29da150cb4132b83b1b22bb14544531a12fba7dac56dd3d28a2c2ec89b7 not found: ID does not exist" Sep 10 16:31:38.179923 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.179870 2562 scope.go:117] "RemoveContainer" containerID="0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631" Sep 10 16:31:38.188128 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.188098 2562 scope.go:117] "RemoveContainer" containerID="0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631" Sep 10 16:31:38.188410 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:31:38.188393 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631\": container with ID starting with 0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631 not found: ID does not exist" containerID="0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631" Sep 10 16:31:38.188468 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.188417 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631"} err="failed to get container status \"0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631\": rpc error: code = NotFound desc = could not find container \"0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631\": container with ID starting with 0fee02830c97069e82fe21c2726b250188aca2a4bdacb4324a5e6547d593b631 not found: ID does not exist" Sep 10 16:31:38.191811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.191766 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" podStartSLOduration=1.191752372 podStartE2EDuration="1.191752372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:38.191205641 +0000 UTC m=+848.917524571" watchObservedRunningTime="2026-09-10 16:31:38.191752372 +0000 UTC m=+848.918071301" Sep 10 16:31:38.205525 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.205490 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6"] Sep 10 16:31:38.209443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.209415 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f89c54f2-55bf8f6ff8-h8hr6"] Sep 10 16:31:38.219884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.219855 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd"] Sep 10 16:31:38.223464 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.223438 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f89c54f2-7c54466f-5xqqd"] Sep 10 16:31:38.387805 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.387723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:38.494128 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.494095 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8"] Sep 10 16:31:38.494368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.494342 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" containerID="cri-o://abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d" gracePeriod=30 Sep 10 16:31:38.504267 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.504234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:38.911215 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:38.911183 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:39.077685 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.077649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:39.178314 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.178234 2562 generic.go:356] "Generic (PLEG): container finished" podID="88f5ea93-2244-489b-8def-75e68bf3da87" containerID="270c7f935832721cd2f4d2fccf2c1026d39910a29a9a67c27978786c57c88848" exitCode=0 Sep 10 16:31:39.178744 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.178320 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" event={"ID":"88f5ea93-2244-489b-8def-75e68bf3da87","Type":"ContainerDied","Data":"270c7f935832721cd2f4d2fccf2c1026d39910a29a9a67c27978786c57c88848"} Sep 10 16:31:39.303657 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.303614 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp"] Sep 10 16:31:39.304154 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.304130 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc90796e-b637-43c2-8005-433f09f00300" containerName="payload-processing" Sep 10 16:31:39.304154 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.304150 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc90796e-b637-43c2-8005-433f09f00300" containerName="payload-processing" Sep 10 16:31:39.304300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.304173 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90794fbd-6817-459d-befc-1676e26cd169" containerName="payload-pre-processing" Sep 10 16:31:39.304300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.304179 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90794fbd-6817-459d-befc-1676e26cd169" containerName="payload-pre-processing" Sep 10 16:31:39.304300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.304251 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="90794fbd-6817-459d-befc-1676e26cd169" containerName="payload-pre-processing" Sep 10 16:31:39.304300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.304263 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc90796e-b637-43c2-8005-433f09f00300" containerName="payload-processing" Sep 10 16:31:39.306233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.306209 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.310295 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.310272 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f39d5ea4-openshift-default-dockercfg-n64s4\"" Sep 10 16:31:39.330973 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.330938 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp"] Sep 10 16:31:39.357758 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.357726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:39.405690 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-certs\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.405895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-socket\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.405895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-data\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.405895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405773 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-envoy\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.405895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405819 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-credential-socket\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.405895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-podinfo\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.406089 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-token\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.406089 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srlx2\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-kube-api-access-srlx2\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.406089 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.405990 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958a7d9-2f96-44d2-ba9e-270cff14017e-istiod-ca-cert\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507057 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-podinfo\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507232 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-token\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507232 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srlx2\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-kube-api-access-srlx2\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507232 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958a7d9-2f96-44d2-ba9e-270cff14017e-istiod-ca-cert\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507232 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-certs\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507232 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-socket\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507494 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-data\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507494 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507271 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-envoy\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507494 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-credential-socket\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507693 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507657 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-certs\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507817 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507737 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-socket\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507817 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-data\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507944 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-credential-socket\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.507984 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.507945 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958a7d9-2f96-44d2-ba9e-270cff14017e-istiod-ca-cert\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.509628 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.509603 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-podinfo\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.509628 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.509617 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-envoy\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.515422 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.515389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-token\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.515551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.515533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srlx2\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-kube-api-access-srlx2\") pod \"e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.617775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.617731 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:39.757539 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.757453 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90794fbd-6817-459d-befc-1676e26cd169" path="/var/lib/kubelet/pods/90794fbd-6817-459d-befc-1676e26cd169/volumes" Sep 10 16:31:39.757920 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.757903 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc90796e-b637-43c2-8005-433f09f00300" path="/var/lib/kubelet/pods/cc90796e-b637-43c2-8005-433f09f00300/volumes" Sep 10 16:31:39.899222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.899191 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp"] Sep 10 16:31:39.901867 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:31:39.901834 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6958a7d9_2f96_44d2_ba9e_270cff14017e.slice/crio-e4fa55daf8589faee54bd14b18ec16e0f2bf916991d44113aa62ab01259fe7aa WatchSource:0}: Error finding container e4fa55daf8589faee54bd14b18ec16e0f2bf916991d44113aa62ab01259fe7aa: Status 404 returned error can't find the container with id e4fa55daf8589faee54bd14b18ec16e0f2bf916991d44113aa62ab01259fe7aa Sep 10 16:31:39.904242 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.904205 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:39.904334 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.904273 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:39.904334 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:39.904303 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:40.003406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.003372 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:40.106803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.106685 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:40.188461 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.188427 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" event={"ID":"6958a7d9-2f96-44d2-ba9e-270cff14017e","Type":"ContainerStarted","Data":"6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841"} Sep 10 16:31:40.188874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.188482 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" event={"ID":"6958a7d9-2f96-44d2-ba9e-270cff14017e","Type":"ContainerStarted","Data":"e4fa55daf8589faee54bd14b18ec16e0f2bf916991d44113aa62ab01259fe7aa"} Sep 10 16:31:40.214226 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.214163 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" podStartSLOduration=1.214139912 podStartE2EDuration="1.214139912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:40.211094704 +0000 UTC m=+850.937413634" watchObservedRunningTime="2026-09-10 16:31:40.214139912 +0000 UTC m=+850.940458869" Sep 10 16:31:40.310693 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.310667 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" Sep 10 16:31:40.415740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.415642 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f5ea93-2244-489b-8def-75e68bf3da87-maas-api-service-ca\") pod \"88f5ea93-2244-489b-8def-75e68bf3da87\" (UID: \"88f5ea93-2244-489b-8def-75e68bf3da87\") " Sep 10 16:31:40.416020 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.415995 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f5ea93-2244-489b-8def-75e68bf3da87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88f5ea93-2244-489b-8def-75e68bf3da87" (UID: "88f5ea93-2244-489b-8def-75e68bf3da87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:31:40.416122 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.416098 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88f5ea93-2244-489b-8def-75e68bf3da87-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:40.618212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.618169 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:40.623904 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.623877 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:40.974106 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:40.974072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:41.197719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:41.197686 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" Sep 10 16:31:41.197719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:41.197704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585" event={"ID":"88f5ea93-2244-489b-8def-75e68bf3da87","Type":"ContainerDied","Data":"eec360634a0ed2c5b475af26a0dc09ad92efbc5932d0ab26cd55cc9af6bbd179"} Sep 10 16:31:41.198194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:41.197764 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eec360634a0ed2c5b475af26a0dc09ad92efbc5932d0ab26cd55cc9af6bbd179" Sep 10 16:31:41.198253 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:41.198199 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:41.199333 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:41.199311 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:31:41.673674 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:41.673631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:41.762316 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:41.762286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:42.641288 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:42.641253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:43.287424 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.287392 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:43.755829 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.755803 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:31:43.855398 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855362 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-data\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855596 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855429 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c936b82f-e678-482f-ac92-cf4d91fb50ce-istiod-ca-cert\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855596 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855454 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-envoy\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855596 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855480 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-token\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855596 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855500 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-certs\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855596 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855518 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-socket\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855596 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855586 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f7k7\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-kube-api-access-2f7k7\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855918 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855607 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-podinfo\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855918 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855648 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-credential-socket\") pod \"c936b82f-e678-482f-ac92-cf4d91fb50ce\" (UID: \"c936b82f-e678-482f-ac92-cf4d91fb50ce\") " Sep 10 16:31:43.855918 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855737 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:43.856126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855920 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:43.856126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855938 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:43.856126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855967 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c936b82f-e678-482f-ac92-cf4d91fb50ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:31:43.856126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.855944 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:43.856332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.856168 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:43.856332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.856189 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c936b82f-e678-482f-ac92-cf4d91fb50ce-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:43.856332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.856202 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:43.856332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.856214 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:43.856332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.856224 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:43.857698 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.857665 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:31:43.857834 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.857772 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:31:43.858367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.858335 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:31:43.858367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.858336 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-kube-api-access-2f7k7" (OuterVolumeSpecName: "kube-api-access-2f7k7") pod "c936b82f-e678-482f-ac92-cf4d91fb50ce" (UID: "c936b82f-e678-482f-ac92-cf4d91fb50ce"). InnerVolumeSpecName "kube-api-access-2f7k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:31:43.957335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.957294 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:43.957335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.957325 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:43.957335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.957334 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f7k7\" (UniqueName: \"kubernetes.io/projected/c936b82f-e678-482f-ac92-cf4d91fb50ce-kube-api-access-2f7k7\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:43.957335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:43.957345 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c936b82f-e678-482f-ac92-cf4d91fb50ce-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:31:44.120276 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.120187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:44.217602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.217567 2562 generic.go:356] "Generic (PLEG): container finished" podID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerID="abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d" exitCode=0 Sep 10 16:31:44.217767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.217621 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" Sep 10 16:31:44.217767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.217652 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" event={"ID":"c936b82f-e678-482f-ac92-cf4d91fb50ce","Type":"ContainerDied","Data":"abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d"} Sep 10 16:31:44.217767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.217707 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8" event={"ID":"c936b82f-e678-482f-ac92-cf4d91fb50ce","Type":"ContainerDied","Data":"3bb13779c5f25389af70dd034a1fdff0384dfd008503f4e0e274a74df055f82d"} Sep 10 16:31:44.217767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.217732 2562 scope.go:117] "RemoveContainer" containerID="abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d" Sep 10 16:31:44.228291 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.228273 2562 scope.go:117] "RemoveContainer" containerID="22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a" Sep 10 16:31:44.236837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.236815 2562 scope.go:117] "RemoveContainer" containerID="abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d" Sep 10 16:31:44.237151 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:31:44.237130 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d\": container with ID starting with abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d not found: ID does not exist" containerID="abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d" Sep 10 16:31:44.237222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.237157 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d"} err="failed to get container status \"abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d\": rpc error: code = NotFound desc = could not find container \"abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d\": container with ID starting with abf9c54c5ea0352ec7b7f894efba4f772c6d4350c1d5ed07b8ef8da1ea28780d not found: ID does not exist" Sep 10 16:31:44.237222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.237175 2562 scope.go:117] "RemoveContainer" containerID="22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a" Sep 10 16:31:44.237452 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:31:44.237435 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a\": container with ID starting with 22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a not found: ID does not exist" containerID="22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a" Sep 10 16:31:44.237494 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.237457 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a"} err="failed to get container status \"22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a\": rpc error: code = NotFound desc = could not find container \"22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a\": container with ID starting with 22dccba29e6f3c40219e4a1a67ded65d7e13bfe044e4f0099311c23cf3546f6a not found: ID does not exist" Sep 10 16:31:44.242332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.242311 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8"] Sep 10 16:31:44.246296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.246274 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f89c54f2-openshift-default-59948847bb-tr4z8"] Sep 10 16:31:44.488664 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.488575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:44.813873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.813836 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:44.911068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:44.911014 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:45.338661 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.338630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:45.439331 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.439302 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:45.510756 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.510727 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:45.514763 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.514735 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64"] Sep 10 16:31:45.515194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.515179 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" Sep 10 16:31:45.515244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.515195 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" Sep 10 16:31:45.515244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.515225 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f5ea93-2244-489b-8def-75e68bf3da87" containerName="revoke-keys" Sep 10 16:31:45.515244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.515232 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f5ea93-2244-489b-8def-75e68bf3da87" containerName="revoke-keys" Sep 10 16:31:45.515345 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.515297 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" Sep 10 16:31:45.515345 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.515307 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f5ea93-2244-489b-8def-75e68bf3da87" containerName="revoke-keys" Sep 10 16:31:45.515345 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.515320 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" Sep 10 16:31:45.518607 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.518591 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:45.522171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.522151 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f39d5ea4-dockercfg-tpqmg\"" Sep 10 16:31:45.522554 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.522534 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f39d5ea4\"" Sep 10 16:31:45.528473 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.528447 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64"] Sep 10 16:31:45.565402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.565367 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6"] Sep 10 16:31:45.565803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.565790 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" Sep 10 16:31:45.565860 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.565804 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" containerName="istio-proxy" Sep 10 16:31:45.568369 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.568345 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:45.579866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.579836 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6"] Sep 10 16:31:45.675540 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.675452 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0451df7d-37ea-4021-81bf-5061cc2ce3d4-plugins-config-volume\") pod \"payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6\" (UID: \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\") " pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:45.675540 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.675494 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mq45\" (UniqueName: \"kubernetes.io/projected/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-kube-api-access-4mq45\") pod \"payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64\" (UID: \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:45.675738 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.675604 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcx9x\" (UniqueName: \"kubernetes.io/projected/0451df7d-37ea-4021-81bf-5061cc2ce3d4-kube-api-access-vcx9x\") pod \"payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6\" (UID: \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\") " pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:45.675738 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.675622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64\" (UID: \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:45.758552 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.758512 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c936b82f-e678-482f-ac92-cf4d91fb50ce" path="/var/lib/kubelet/pods/c936b82f-e678-482f-ac92-cf4d91fb50ce/volumes" Sep 10 16:31:45.776948 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.776900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0451df7d-37ea-4021-81bf-5061cc2ce3d4-plugins-config-volume\") pod \"payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6\" (UID: \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\") " pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:45.777181 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.776957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mq45\" (UniqueName: \"kubernetes.io/projected/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-kube-api-access-4mq45\") pod \"payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64\" (UID: \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:45.777181 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.777116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcx9x\" (UniqueName: \"kubernetes.io/projected/0451df7d-37ea-4021-81bf-5061cc2ce3d4-kube-api-access-vcx9x\") pod \"payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6\" (UID: \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\") " pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:45.777181 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.777143 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64\" (UID: \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:45.777666 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.777642 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0451df7d-37ea-4021-81bf-5061cc2ce3d4-plugins-config-volume\") pod \"payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6\" (UID: \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\") " pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:45.777878 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.777858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64\" (UID: \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:45.785898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.785867 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcx9x\" (UniqueName: \"kubernetes.io/projected/0451df7d-37ea-4021-81bf-5061cc2ce3d4-kube-api-access-vcx9x\") pod \"payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6\" (UID: \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\") " pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:45.786026 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.785944 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mq45\" (UniqueName: \"kubernetes.io/projected/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-kube-api-access-4mq45\") pod \"payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64\" (UID: \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:45.830313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.830277 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:45.879261 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:45.879224 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:46.104228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.104203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64"] Sep 10 16:31:46.106233 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:31:46.106205 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d627e5_bcb8_4d4e_becb_f58b3653c9b5.slice/crio-87381250d42b40b5f14d5c8ca3a84eb553e8be2ac19072f08ac4a118dbbd81d5 WatchSource:0}: Error finding container 87381250d42b40b5f14d5c8ca3a84eb553e8be2ac19072f08ac4a118dbbd81d5: Status 404 returned error can't find the container with id 87381250d42b40b5f14d5c8ca3a84eb553e8be2ac19072f08ac4a118dbbd81d5 Sep 10 16:31:46.162015 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.161986 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6"] Sep 10 16:31:46.163708 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:31:46.163677 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0451df7d_37ea_4021_81bf_5061cc2ce3d4.slice/crio-3a85456dc00ebfe6e6cf3463a8fc1c7e2a2db8fc84933b0d2bcbc161fa0e4c3d WatchSource:0}: Error finding container 3a85456dc00ebfe6e6cf3463a8fc1c7e2a2db8fc84933b0d2bcbc161fa0e4c3d: Status 404 returned error can't find the container with id 3a85456dc00ebfe6e6cf3463a8fc1c7e2a2db8fc84933b0d2bcbc161fa0e4c3d Sep 10 16:31:46.233570 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.233535 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" event={"ID":"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5","Type":"ContainerStarted","Data":"8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c"} Sep 10 16:31:46.233755 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.233600 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" event={"ID":"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5","Type":"ContainerStarted","Data":"87381250d42b40b5f14d5c8ca3a84eb553e8be2ac19072f08ac4a118dbbd81d5"} Sep 10 16:31:46.233755 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.233629 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:46.235303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.235276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" event={"ID":"0451df7d-37ea-4021-81bf-5061cc2ce3d4","Type":"ContainerStarted","Data":"51dc0b1b0879ad2d3f9cf63e8ab13f83e9b8896f608e1fbf692434b5072d71b0"} Sep 10 16:31:46.235418 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.235324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" event={"ID":"0451df7d-37ea-4021-81bf-5061cc2ce3d4","Type":"ContainerStarted","Data":"3a85456dc00ebfe6e6cf3463a8fc1c7e2a2db8fc84933b0d2bcbc161fa0e4c3d"} Sep 10 16:31:46.235418 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.235347 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:46.254091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.254014 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" podStartSLOduration=1.253997931 podStartE2EDuration="1.253997931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:46.250891757 +0000 UTC m=+856.977210714" watchObservedRunningTime="2026-09-10 16:31:46.253997931 +0000 UTC m=+856.980316861" Sep 10 16:31:46.269759 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:46.269707 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" podStartSLOduration=1.269690267 podStartE2EDuration="1.269690267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:46.267392487 +0000 UTC m=+856.993711434" watchObservedRunningTime="2026-09-10 16:31:46.269690267 +0000 UTC m=+856.996009197" Sep 10 16:31:47.354505 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:47.354472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:47.984967 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:47.984936 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:48.086798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:48.086768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:48.187535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:48.187489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:48.246722 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:48.246677 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585"] Sep 10 16:31:48.248834 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:48.248806 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2d818c-8p585"] Sep 10 16:31:48.261071 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:48.261013 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:48.603097 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:48.602977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:48.706639 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:48.706596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:49.445228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:49.445194 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:49.757478 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:49.757448 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f5ea93-2244-489b-8def-75e68bf3da87" path="/var/lib/kubelet/pods/88f5ea93-2244-489b-8def-75e68bf3da87/volumes" Sep 10 16:31:50.078846 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:50.078764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:50.890791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:50.890759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:52.755239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:52.755196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:31:55.352245 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:55.352210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:55.454790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:55.454746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:55.604902 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:55.604825 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk"] Sep 10 16:31:55.605085 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:55.605062 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" containerID="cri-o://6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898" gracePeriod=30 Sep 10 16:31:55.632455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:55.632410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:56.048161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.048084 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:56.314482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.314399 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf"] Sep 10 16:31:56.317195 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.317172 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.320124 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.320106 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-09da25-openshift-default-dockercfg-725gr\"" Sep 10 16:31:56.338112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.337435 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf"] Sep 10 16:31:56.376993 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.376959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:56.465782 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.465745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:56.494854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.494809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cb139d-fdb3-4eea-a171-7041fd052503-istio-podinfo\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.495071 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.494884 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-socket\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.495071 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.494906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-data\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.495071 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.495051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-istio-token\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.495236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.495111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5lb\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-kube-api-access-vs5lb\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.495236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.495202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-credential-socket\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.495236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.495219 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-certs\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.495347 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.495291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cb139d-fdb3-4eea-a171-7041fd052503-istiod-ca-cert\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.495347 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.495313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-envoy\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.595898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.595799 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5lb\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-kube-api-access-vs5lb\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.595898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.595872 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-credential-socket\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.595898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.595889 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-certs\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.595924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cb139d-fdb3-4eea-a171-7041fd052503-istiod-ca-cert\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.595944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-envoy\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.595979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cb139d-fdb3-4eea-a171-7041fd052503-istio-podinfo\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.596009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-socket\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.596033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-data\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.596130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-istio-token\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.596344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-credential-socket\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.596459 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-certs\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596725 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.596701 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-data\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.596728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-socket\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.596968 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.596942 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cb139d-fdb3-4eea-a171-7041fd052503-istiod-ca-cert\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.598474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.598446 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-envoy\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.598579 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.598529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cb139d-fdb3-4eea-a171-7041fd052503-istio-podinfo\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.604303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.604280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-istio-token\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.604594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.604573 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5lb\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-kube-api-access-vs5lb\") pod \"e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.628078 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.628046 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:56.933529 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.933498 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf"] Sep 10 16:31:56.936058 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:31:56.936014 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cb139d_fdb3_4eea_a171_7041fd052503.slice/crio-0289fd72e7d1c8d4545b7402c45e22cbc3582818771c638d4fe41eb2bad8d698 WatchSource:0}: Error finding container 0289fd72e7d1c8d4545b7402c45e22cbc3582818771c638d4fe41eb2bad8d698: Status 404 returned error can't find the container with id 0289fd72e7d1c8d4545b7402c45e22cbc3582818771c638d4fe41eb2bad8d698 Sep 10 16:31:56.938886 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.938851 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:56.938969 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.938928 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:56.938969 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:56.938960 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:31:57.246980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:57.246928 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:31:57.248847 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:57.248202 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:31:57.305479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:57.305437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" event={"ID":"37cb139d-fdb3-4eea-a171-7041fd052503","Type":"ContainerStarted","Data":"23812e6cd5632a22890834b69cc394925aef9b1ebc084374b5b400063eb84175"} Sep 10 16:31:57.305672 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:57.305518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" event={"ID":"37cb139d-fdb3-4eea-a171-7041fd052503","Type":"ContainerStarted","Data":"0289fd72e7d1c8d4545b7402c45e22cbc3582818771c638d4fe41eb2bad8d698"} Sep 10 16:31:57.330424 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:57.330359 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podStartSLOduration=1.330341537 podStartE2EDuration="1.330341537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:31:57.327419923 +0000 UTC m=+868.053738851" watchObservedRunningTime="2026-09-10 16:31:57.330341537 +0000 UTC m=+868.056660466" Sep 10 16:31:57.628919 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:57.628831 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:58.634099 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:58.634066 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:59.290712 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:59.290635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:31:59.318840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:59.318800 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:59.319811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:59.319790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:31:59.412989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:31:59.412957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:00.166062 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.161820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:00.915591 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.915567 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:32:00.946504 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946465 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istiod-ca-cert\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946683 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946524 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-credential-socket\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946683 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946554 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nvp7\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-kube-api-access-4nvp7\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946683 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946643 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-certs\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946683 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946672 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-token\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946929 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946720 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-envoy\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946929 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946783 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-data\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946929 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946822 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-socket\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946929 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946872 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-podinfo\") pod \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\" (UID: \"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7\") " Sep 10 16:32:00.946929 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.946895 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:32:00.947337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.947315 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:00.947489 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.947330 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:00.947810 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.947551 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:00.947810 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.947778 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:00.948475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.948440 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:00.950331 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.950174 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:00.950652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.950629 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:32:00.950851 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.950828 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:32:00.951394 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:00.951373 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-kube-api-access-4nvp7" (OuterVolumeSpecName: "kube-api-access-4nvp7") pod "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" (UID: "0a74b5c2-a1f0-40a4-bb2d-3a27390925a7"). InnerVolumeSpecName "kube-api-access-4nvp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:32:01.048814 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.048783 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:01.048814 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.048810 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:01.048982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.048819 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:01.048982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.048828 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:01.048982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.048837 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvp7\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-kube-api-access-4nvp7\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:01.048982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.048845 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:01.048982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.048854 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:01.048982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.048862 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:01.071278 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.071200 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:01.331735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.331696 2562 generic.go:356] "Generic (PLEG): container finished" podID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerID="6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898" exitCode=0 Sep 10 16:32:01.332188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.331767 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" Sep 10 16:32:01.332188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.331774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" event={"ID":"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7","Type":"ContainerDied","Data":"6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898"} Sep 10 16:32:01.332188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.331854 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk" event={"ID":"0a74b5c2-a1f0-40a4-bb2d-3a27390925a7","Type":"ContainerDied","Data":"3c6ddf3ac2b12bd524bceaa4875297a272cc0bdd82b9f276f9a5554df919e1d2"} Sep 10 16:32:01.332188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.331876 2562 scope.go:117] "RemoveContainer" containerID="6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898" Sep 10 16:32:01.341074 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.341053 2562 scope.go:117] "RemoveContainer" containerID="ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb" Sep 10 16:32:01.350622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.350600 2562 scope.go:117] "RemoveContainer" containerID="6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898" Sep 10 16:32:01.350966 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:32:01.350940 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898\": container with ID starting with 6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898 not found: ID does not exist" containerID="6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898" Sep 10 16:32:01.351108 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.350979 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898"} err="failed to get container status \"6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898\": rpc error: code = NotFound desc = could not find container \"6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898\": container with ID starting with 6983e3bb1e9b8e9f4d107f138b291df83df134018a7c7999a7a3d114ea4ba898 not found: ID does not exist" Sep 10 16:32:01.351108 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.351003 2562 scope.go:117] "RemoveContainer" containerID="ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb" Sep 10 16:32:01.351377 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:32:01.351353 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb\": container with ID starting with ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb not found: ID does not exist" containerID="ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb" Sep 10 16:32:01.351438 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.351391 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb"} err="failed to get container status \"ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb\": rpc error: code = NotFound desc = could not find container \"ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb\": container with ID starting with ccb4663934bc2f226ef492318eb2acd5f051ccbe30cc0ffd37bdab56421455eb not found: ID does not exist" Sep 10 16:32:01.363942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.363908 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk"] Sep 10 16:32:01.371992 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.371962 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-2d818c-openshift-default-5856789fd-57jbk"] Sep 10 16:32:01.378533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.378503 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz"] Sep 10 16:32:01.379196 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.379178 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" Sep 10 16:32:01.379258 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.379200 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" Sep 10 16:32:01.379258 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.379216 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" Sep 10 16:32:01.379258 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.379225 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" Sep 10 16:32:01.379359 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.379333 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" Sep 10 16:32:01.379359 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.379348 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" containerName="istio-proxy" Sep 10 16:32:01.382508 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.382490 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" Sep 10 16:32:01.385379 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.385357 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:32:01.385473 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.385402 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:32:01.390776 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.390747 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz"] Sep 10 16:32:01.453881 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.453838 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1e536-ed01-49a9-8edc-5bcd8813b7d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz\" (UID: \"dbf1e536-ed01-49a9-8edc-5bcd8813b7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" Sep 10 16:32:01.555108 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.555032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1e536-ed01-49a9-8edc-5bcd8813b7d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz\" (UID: \"dbf1e536-ed01-49a9-8edc-5bcd8813b7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" Sep 10 16:32:01.555879 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.555855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1e536-ed01-49a9-8edc-5bcd8813b7d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz\" (UID: \"dbf1e536-ed01-49a9-8edc-5bcd8813b7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" Sep 10 16:32:01.697056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.696953 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" Sep 10 16:32:01.761340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.760567 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a74b5c2-a1f0-40a4-bb2d-3a27390925a7" path="/var/lib/kubelet/pods/0a74b5c2-a1f0-40a4-bb2d-3a27390925a7/volumes" Sep 10 16:32:01.989862 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:01.989836 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz"] Sep 10 16:32:01.992313 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:32:01.992282 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf1e536_ed01_49a9_8edc_5bcd8813b7d2.slice/crio-dd3aba612d94707d2e8e962b54642b23e58807f8ad8c4634b27a951e1756c65c WatchSource:0}: Error finding container dd3aba612d94707d2e8e962b54642b23e58807f8ad8c4634b27a951e1756c65c: Status 404 returned error can't find the container with id dd3aba612d94707d2e8e962b54642b23e58807f8ad8c4634b27a951e1756c65c Sep 10 16:32:02.192730 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:02.192691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:02.339717 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:02.339670 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" event={"ID":"dbf1e536-ed01-49a9-8edc-5bcd8813b7d2","Type":"ContainerStarted","Data":"dd3aba612d94707d2e8e962b54642b23e58807f8ad8c4634b27a951e1756c65c"} Sep 10 16:32:03.081308 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.081275 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:03.182735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.182646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:03.348358 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.348318 2562 generic.go:356] "Generic (PLEG): container finished" podID="dbf1e536-ed01-49a9-8edc-5bcd8813b7d2" containerID="ee3ff8b8d91b372b1bb62e676cfea6dd320166b6766f7e8144e5609c62486ffa" exitCode=0 Sep 10 16:32:03.348791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.348389 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" event={"ID":"dbf1e536-ed01-49a9-8edc-5bcd8813b7d2","Type":"ContainerDied","Data":"ee3ff8b8d91b372b1bb62e676cfea6dd320166b6766f7e8144e5609c62486ffa"} Sep 10 16:32:03.406192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.406158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:03.561782 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.561752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:03.660010 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.659971 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:03.678764 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.678730 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z"] Sep 10 16:32:03.681210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.681192 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:03.684411 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.684385 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-09da25-dockercfg-wbxk4\"" Sep 10 16:32:03.684411 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.684386 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-09da25\"" Sep 10 16:32:03.694729 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.694694 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z"] Sep 10 16:32:03.785564 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.785499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkqm\" (UniqueName: \"kubernetes.io/projected/70539d7e-de24-4c4b-aece-2ebbb07554c2-kube-api-access-8lkqm\") pod \"payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z\" (UID: \"70539d7e-de24-4c4b-aece-2ebbb07554c2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:03.785745 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.785592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70539d7e-de24-4c4b-aece-2ebbb07554c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z\" (UID: \"70539d7e-de24-4c4b-aece-2ebbb07554c2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:03.886705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.886611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkqm\" (UniqueName: \"kubernetes.io/projected/70539d7e-de24-4c4b-aece-2ebbb07554c2-kube-api-access-8lkqm\") pod \"payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z\" (UID: \"70539d7e-de24-4c4b-aece-2ebbb07554c2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:03.886705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.886671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70539d7e-de24-4c4b-aece-2ebbb07554c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z\" (UID: \"70539d7e-de24-4c4b-aece-2ebbb07554c2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:03.887407 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.887387 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70539d7e-de24-4c4b-aece-2ebbb07554c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z\" (UID: \"70539d7e-de24-4c4b-aece-2ebbb07554c2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:03.895932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.895904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkqm\" (UniqueName: \"kubernetes.io/projected/70539d7e-de24-4c4b-aece-2ebbb07554c2-kube-api-access-8lkqm\") pod \"payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z\" (UID: \"70539d7e-de24-4c4b-aece-2ebbb07554c2\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:03.992974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:03.992938 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:04.073652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.073617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:04.266888 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.266859 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z"] Sep 10 16:32:04.269252 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:32:04.269223 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70539d7e_de24_4c4b_aece_2ebbb07554c2.slice/crio-12fe86f5306505951b4a153311e25cf1c2f51333776e56f31837dae9d284a636 WatchSource:0}: Error finding container 12fe86f5306505951b4a153311e25cf1c2f51333776e56f31837dae9d284a636: Status 404 returned error can't find the container with id 12fe86f5306505951b4a153311e25cf1c2f51333776e56f31837dae9d284a636 Sep 10 16:32:04.357344 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.357303 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" event={"ID":"70539d7e-de24-4c4b-aece-2ebbb07554c2","Type":"ContainerStarted","Data":"18b6dc8eadda57ffd401e2cdd45bf1d563c250a02fd9101768ba4a6dbe275edb"} Sep 10 16:32:04.357821 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.357375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" event={"ID":"70539d7e-de24-4c4b-aece-2ebbb07554c2","Type":"ContainerStarted","Data":"12fe86f5306505951b4a153311e25cf1c2f51333776e56f31837dae9d284a636"} Sep 10 16:32:04.357821 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.357410 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:04.359444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.359413 2562 generic.go:356] "Generic (PLEG): container finished" podID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerID="6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841" exitCode=0 Sep 10 16:32:04.359564 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.359490 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" event={"ID":"6958a7d9-2f96-44d2-ba9e-270cff14017e","Type":"ContainerDied","Data":"6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841"} Sep 10 16:32:04.360135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.360120 2562 scope.go:117] "RemoveContainer" containerID="6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841" Sep 10 16:32:04.361550 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.361517 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:32:04.361650 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.361590 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:32:04.361650 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.361624 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:32:04.380103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.379600 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" podStartSLOduration=1.379576119 podStartE2EDuration="1.379576119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:32:04.375139233 +0000 UTC m=+875.101458164" watchObservedRunningTime="2026-09-10 16:32:04.379576119 +0000 UTC m=+875.105895050" Sep 10 16:32:04.516180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.515750 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" Sep 10 16:32:04.595549 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.595450 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1e536-ed01-49a9-8edc-5bcd8813b7d2-maas-api-service-ca\") pod \"dbf1e536-ed01-49a9-8edc-5bcd8813b7d2\" (UID: \"dbf1e536-ed01-49a9-8edc-5bcd8813b7d2\") " Sep 10 16:32:04.595823 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.595795 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf1e536-ed01-49a9-8edc-5bcd8813b7d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbf1e536-ed01-49a9-8edc-5bcd8813b7d2" (UID: "dbf1e536-ed01-49a9-8edc-5bcd8813b7d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:32:04.696947 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.696909 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1e536-ed01-49a9-8edc-5bcd8813b7d2-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:04.794854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:04.794809 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:05.371296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.371246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" event={"ID":"dbf1e536-ed01-49a9-8edc-5bcd8813b7d2","Type":"ContainerDied","Data":"dd3aba612d94707d2e8e962b54642b23e58807f8ad8c4634b27a951e1756c65c"} Sep 10 16:32:05.371867 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.371840 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd3aba612d94707d2e8e962b54642b23e58807f8ad8c4634b27a951e1756c65c" Sep 10 16:32:05.372081 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.372068 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz" Sep 10 16:32:05.382265 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.382231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" event={"ID":"6958a7d9-2f96-44d2-ba9e-270cff14017e","Type":"ContainerStarted","Data":"3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624"} Sep 10 16:32:05.473407 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.473369 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:05.618492 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.618455 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:32:05.618769 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.618744 2562 patch_prober.go:28] interesting pod/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 16:32:05.618824 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.618789 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 16:32:05.970065 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:05.966592 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:06.345008 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:06.344974 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:06.618889 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:06.618804 2562 patch_prober.go:28] interesting pod/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 16:32:06.618889 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:06.618863 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 16:32:07.052451 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:07.052394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:07.618458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:07.618412 2562 patch_prober.go:28] interesting pod/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 16:32:07.618671 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:07.618479 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 16:32:08.618281 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:08.618245 2562 patch_prober.go:28] interesting pod/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 16:32:08.618658 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:08.618306 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 16:32:09.461497 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:09.461448 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:32:09.618780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:09.618744 2562 patch_prober.go:28] interesting pod/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 16:32:09.619381 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:09.619344 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 16:32:10.625508 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:10.625470 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:32:10.625985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:10.625896 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:32:10.627737 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:10.627703 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:32:12.017573 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.017545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:12.072512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.072469 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6"] Sep 10 16:32:12.073009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.072972 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" podUID="0451df7d-37ea-4021-81bf-5061cc2ce3d4" containerName="payload-processing" containerID="cri-o://51dc0b1b0879ad2d3f9cf63e8ab13f83e9b8896f608e1fbf692434b5072d71b0" gracePeriod=30 Sep 10 16:32:12.080397 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.080346 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64"] Sep 10 16:32:12.082313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.082278 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" podUID="b3d627e5-bcb8-4d4e-becb-f58b3653c9b5" containerName="payload-pre-processing" containerID="cri-o://8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c" gracePeriod=30 Sep 10 16:32:12.091365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.090832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:12.187325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.187292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:12.213461 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.213206 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz"] Sep 10 16:32:12.216797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.216756 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f39d5ea4-k89mz"] Sep 10 16:32:12.279063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.278857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:12.393991 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.392089 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp"] Sep 10 16:32:12.397421 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.397388 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:32:12.404192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.404158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:12.430004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.429119 2562 generic.go:356] "Generic (PLEG): container finished" podID="b3d627e5-bcb8-4d4e-becb-f58b3653c9b5" containerID="8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c" exitCode=0 Sep 10 16:32:12.430004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.429293 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" event={"ID":"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5","Type":"ContainerDied","Data":"8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c"} Sep 10 16:32:12.430004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.429354 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" event={"ID":"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5","Type":"ContainerDied","Data":"87381250d42b40b5f14d5c8ca3a84eb553e8be2ac19072f08ac4a118dbbd81d5"} Sep 10 16:32:12.430004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.429386 2562 scope.go:117] "RemoveContainer" containerID="8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c" Sep 10 16:32:12.430004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.429455 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64" Sep 10 16:32:12.431828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.431797 2562 generic.go:356] "Generic (PLEG): container finished" podID="0451df7d-37ea-4021-81bf-5061cc2ce3d4" containerID="51dc0b1b0879ad2d3f9cf63e8ab13f83e9b8896f608e1fbf692434b5072d71b0" exitCode=0 Sep 10 16:32:12.432080 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.431837 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" event={"ID":"0451df7d-37ea-4021-81bf-5061cc2ce3d4","Type":"ContainerDied","Data":"51dc0b1b0879ad2d3f9cf63e8ab13f83e9b8896f608e1fbf692434b5072d71b0"} Sep 10 16:32:12.432459 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.432440 2562 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" secret="" err="secret \"e2e-derive-f39d5ea4-openshift-default-dockercfg-n64s4\" not found" Sep 10 16:32:12.457290 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.446616 2562 scope.go:117] "RemoveContainer" containerID="8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c" Sep 10 16:32:12.457290 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:32:12.446994 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c\": container with ID starting with 8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c not found: ID does not exist" containerID="8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c" Sep 10 16:32:12.457290 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.447031 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c"} err="failed to get container status \"8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c\": rpc error: code = NotFound desc = could not find container \"8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c\": container with ID starting with 8839a89d5c86db419d2ec7857a2be51e4d322b760549723d856663e9327bc28c not found: ID does not exist" Sep 10 16:32:12.498765 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.498730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-plugins-config-volume\") pod \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\" (UID: \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\") " Sep 10 16:32:12.498887 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.498853 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mq45\" (UniqueName: \"kubernetes.io/projected/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-kube-api-access-4mq45\") pod \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\" (UID: \"b3d627e5-bcb8-4d4e-becb-f58b3653c9b5\") " Sep 10 16:32:12.499798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.499770 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3d627e5-bcb8-4d4e-becb-f58b3653c9b5" (UID: "b3d627e5-bcb8-4d4e-becb-f58b3653c9b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:32:12.503220 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.503188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-kube-api-access-4mq45" (OuterVolumeSpecName: "kube-api-access-4mq45") pod "b3d627e5-bcb8-4d4e-becb-f58b3653c9b5" (UID: "b3d627e5-bcb8-4d4e-becb-f58b3653c9b5"). InnerVolumeSpecName "kube-api-access-4mq45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:32:12.515560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.515531 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:32:12.599981 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.599889 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0451df7d-37ea-4021-81bf-5061cc2ce3d4-plugins-config-volume\") pod \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\" (UID: \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\") " Sep 10 16:32:12.600152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.600012 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcx9x\" (UniqueName: \"kubernetes.io/projected/0451df7d-37ea-4021-81bf-5061cc2ce3d4-kube-api-access-vcx9x\") pod \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\" (UID: \"0451df7d-37ea-4021-81bf-5061cc2ce3d4\") " Sep 10 16:32:12.600392 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.600346 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0451df7d-37ea-4021-81bf-5061cc2ce3d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0451df7d-37ea-4021-81bf-5061cc2ce3d4" (UID: "0451df7d-37ea-4021-81bf-5061cc2ce3d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:32:12.600636 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.600612 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0451df7d-37ea-4021-81bf-5061cc2ce3d4-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:12.600779 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.600755 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mq45\" (UniqueName: \"kubernetes.io/projected/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-kube-api-access-4mq45\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:12.600832 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.600789 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:12.602333 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.602311 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0451df7d-37ea-4021-81bf-5061cc2ce3d4-kube-api-access-vcx9x" (OuterVolumeSpecName: "kube-api-access-vcx9x") pod "0451df7d-37ea-4021-81bf-5061cc2ce3d4" (UID: "0451df7d-37ea-4021-81bf-5061cc2ce3d4"). InnerVolumeSpecName "kube-api-access-vcx9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:32:12.629704 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.629669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:12.701582 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.701549 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcx9x\" (UniqueName: \"kubernetes.io/projected/0451df7d-37ea-4021-81bf-5061cc2ce3d4-kube-api-access-vcx9x\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:12.759572 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.759531 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64"] Sep 10 16:32:12.760202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:12.760180 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f39d5ea4-7c9d9bf6f4-phc64"] Sep 10 16:32:13.080774 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.080733 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:13.173758 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.173723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:13.274016 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.273978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:13.449876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.449762 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" containerID="cri-o://3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624" gracePeriod=30 Sep 10 16:32:13.450253 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.450229 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" Sep 10 16:32:13.453156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.453117 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6" event={"ID":"0451df7d-37ea-4021-81bf-5061cc2ce3d4","Type":"ContainerDied","Data":"3a85456dc00ebfe6e6cf3463a8fc1c7e2a2db8fc84933b0d2bcbc161fa0e4c3d"} Sep 10 16:32:13.453793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.453772 2562 scope.go:117] "RemoveContainer" containerID="51dc0b1b0879ad2d3f9cf63e8ab13f83e9b8896f608e1fbf692434b5072d71b0" Sep 10 16:32:13.509410 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.509373 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6"] Sep 10 16:32:13.517306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.517274 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f39d5ea4-d74b5d77-gvdf6"] Sep 10 16:32:13.680120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.680063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:13.763340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.763298 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0451df7d-37ea-4021-81bf-5061cc2ce3d4" path="/var/lib/kubelet/pods/0451df7d-37ea-4021-81bf-5061cc2ce3d4/volumes" Sep 10 16:32:13.763939 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.763912 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3d627e5-bcb8-4d4e-becb-f58b3653c9b5" path="/var/lib/kubelet/pods/b3d627e5-bcb8-4d4e-becb-f58b3653c9b5/volumes" Sep 10 16:32:13.764411 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.764385 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf1e536-ed01-49a9-8edc-5bcd8813b7d2" path="/var/lib/kubelet/pods/dbf1e536-ed01-49a9-8edc-5bcd8813b7d2/volumes" Sep 10 16:32:13.989983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:13.989948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:14.457330 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:14.457226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:14.642145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:14.642104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:15.161863 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:15.161765 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:15.384798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:15.384769 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:32:15.863965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:15.863931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:16.472133 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:16.472090 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:18.715602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.715573 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:32:18.865743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.865704 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-socket\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.865924 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.865760 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-envoy\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.865924 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.865785 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958a7d9-2f96-44d2-ba9e-270cff14017e-istiod-ca-cert\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.865924 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.865835 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-token\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.866112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.865940 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-data\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.866112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.865993 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-certs\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.866112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866026 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-podinfo\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.866112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866032 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:18.866323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866113 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-credential-socket\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.866323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866141 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srlx2\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-kube-api-access-srlx2\") pod \"6958a7d9-2f96-44d2-ba9e-270cff14017e\" (UID: \"6958a7d9-2f96-44d2-ba9e-270cff14017e\") " Sep 10 16:32:18.866323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866213 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958a7d9-2f96-44d2-ba9e-270cff14017e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:32:18.866323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866284 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:18.866526 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866418 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:18.866526 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866481 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:18.866650 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866628 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:18.866714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866651 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6958a7d9-2f96-44d2-ba9e-270cff14017e-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:18.866714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866666 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:18.866714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866679 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:18.866714 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.866692 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:18.868103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.868079 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:32:18.868697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.868669 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:32:18.868697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.868688 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:32:18.868836 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.868706 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-kube-api-access-srlx2" (OuterVolumeSpecName: "kube-api-access-srlx2") pod "6958a7d9-2f96-44d2-ba9e-270cff14017e" (UID: "6958a7d9-2f96-44d2-ba9e-270cff14017e"). InnerVolumeSpecName "kube-api-access-srlx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:32:18.967495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.967453 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:18.967495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.967487 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srlx2\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-kube-api-access-srlx2\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:18.967495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.967500 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:18.967495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:18.967508 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6958a7d9-2f96-44d2-ba9e-270cff14017e-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:19.499177 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.499139 2562 generic.go:356] "Generic (PLEG): container finished" podID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerID="3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624" exitCode=0 Sep 10 16:32:19.499389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.499220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" event={"ID":"6958a7d9-2f96-44d2-ba9e-270cff14017e","Type":"ContainerDied","Data":"3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624"} Sep 10 16:32:19.499389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.499262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" event={"ID":"6958a7d9-2f96-44d2-ba9e-270cff14017e","Type":"ContainerDied","Data":"e4fa55daf8589faee54bd14b18ec16e0f2bf916991d44113aa62ab01259fe7aa"} Sep 10 16:32:19.499389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.499284 2562 scope.go:117] "RemoveContainer" containerID="3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624" Sep 10 16:32:19.499389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.499385 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp" Sep 10 16:32:19.503116 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.503078 2562 generic.go:356] "Generic (PLEG): container finished" podID="37cb139d-fdb3-4eea-a171-7041fd052503" containerID="23812e6cd5632a22890834b69cc394925aef9b1ebc084374b5b400063eb84175" exitCode=0 Sep 10 16:32:19.503261 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.503155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" event={"ID":"37cb139d-fdb3-4eea-a171-7041fd052503","Type":"ContainerDied","Data":"23812e6cd5632a22890834b69cc394925aef9b1ebc084374b5b400063eb84175"} Sep 10 16:32:19.503753 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.503728 2562 scope.go:117] "RemoveContainer" containerID="23812e6cd5632a22890834b69cc394925aef9b1ebc084374b5b400063eb84175" Sep 10 16:32:19.505219 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.505182 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:32:19.505319 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.505276 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:32:19.505374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.505327 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:32:19.542837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.542807 2562 scope.go:117] "RemoveContainer" containerID="6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841" Sep 10 16:32:19.556522 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.556498 2562 scope.go:117] "RemoveContainer" containerID="3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624" Sep 10 16:32:19.556931 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:32:19.556904 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624\": container with ID starting with 3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624 not found: ID does not exist" containerID="3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624" Sep 10 16:32:19.557032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.556945 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624"} err="failed to get container status \"3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624\": rpc error: code = NotFound desc = could not find container \"3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624\": container with ID starting with 3cb6ca51a127ed4690d8e5bff40f424ab8e7f17ac3de00a663213ac5c144d624 not found: ID does not exist" Sep 10 16:32:19.557032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.556971 2562 scope.go:117] "RemoveContainer" containerID="6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841" Sep 10 16:32:19.557365 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:32:19.557343 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841\": container with ID starting with 6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841 not found: ID does not exist" containerID="6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841" Sep 10 16:32:19.557487 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.557468 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841"} err="failed to get container status \"6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841\": rpc error: code = NotFound desc = could not find container \"6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841\": container with ID starting with 6d96518d96087955dfb949c4edfbc04b9c83d4cbe7671418343a47a7ace47841 not found: ID does not exist" Sep 10 16:32:19.564930 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.564882 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp"] Sep 10 16:32:19.567178 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.567156 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f39d5ea4-openshift-default-7b97f4f55b-zwbnp"] Sep 10 16:32:19.761445 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:19.761407 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" path="/var/lib/kubelet/pods/6958a7d9-2f96-44d2-ba9e-270cff14017e/volumes" Sep 10 16:32:20.511048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:20.511000 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" event={"ID":"37cb139d-fdb3-4eea-a171-7041fd052503","Type":"ContainerStarted","Data":"b05c067714c2b19a3be2136457b38fc193e07375335974720c3997dc8d5478ee"} Sep 10 16:32:20.628986 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:20.628949 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:32:20.629217 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:20.629194 2562 patch_prober.go:28] interesting pod/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 16:32:20.629265 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:20.629248 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 16:32:20.695531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:20.695493 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:21.629470 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:21.629429 2562 patch_prober.go:28] interesting pod/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 16:32:21.629880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:21.629488 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 16:32:22.391876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:22.391841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:22.629138 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:22.629104 2562 patch_prober.go:28] interesting pod/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 16:32:22.629332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:22.629180 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 16:32:23.573585 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:23.573542 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:23.629442 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:23.629409 2562 patch_prober.go:28] interesting pod/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 16:32:23.629613 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:23.629465 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 16:32:23.933521 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:23.933433 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:24.629123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:24.629083 2562 patch_prober.go:28] interesting pod/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 16:32:24.629518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:24.629145 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 16:32:25.234249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:25.234209 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:32:25.629103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:25.629061 2562 patch_prober.go:28] interesting pod/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 10 16:32:25.629289 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:25.629125 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 10 16:32:26.632482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:26.632450 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:32:26.632858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:26.632713 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:32:26.633365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:26.633347 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:32:29.743419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:29.743389 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:32:29.743896 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:29.743881 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:32:39.249455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249418 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h"] Sep 10 16:32:39.249848 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249833 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d627e5-bcb8-4d4e-becb-f58b3653c9b5" containerName="payload-pre-processing" Sep 10 16:32:39.249848 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249844 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d627e5-bcb8-4d4e-becb-f58b3653c9b5" containerName="payload-pre-processing" Sep 10 16:32:39.249934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249858 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" Sep 10 16:32:39.249934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249863 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" Sep 10 16:32:39.249934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249870 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" Sep 10 16:32:39.249934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249875 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" Sep 10 16:32:39.249934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249893 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0451df7d-37ea-4021-81bf-5061cc2ce3d4" containerName="payload-processing" Sep 10 16:32:39.249934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249898 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0451df7d-37ea-4021-81bf-5061cc2ce3d4" containerName="payload-processing" Sep 10 16:32:39.249934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249908 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf1e536-ed01-49a9-8edc-5bcd8813b7d2" containerName="revoke-keys" Sep 10 16:32:39.249934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249913 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf1e536-ed01-49a9-8edc-5bcd8813b7d2" containerName="revoke-keys" Sep 10 16:32:39.250259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249977 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d627e5-bcb8-4d4e-becb-f58b3653c9b5" containerName="payload-pre-processing" Sep 10 16:32:39.250259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249985 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf1e536-ed01-49a9-8edc-5bcd8813b7d2" containerName="revoke-keys" Sep 10 16:32:39.250259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.249994 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" Sep 10 16:32:39.250259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.250000 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0451df7d-37ea-4021-81bf-5061cc2ce3d4" containerName="payload-processing" Sep 10 16:32:39.250259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.250007 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6958a7d9-2f96-44d2-ba9e-270cff14017e" containerName="istio-proxy" Sep 10 16:32:39.253278 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.253260 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.256465 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.256440 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-09da25\"/\"kube-root-ca.crt\"" Sep 10 16:32:39.258112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.258088 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-09da25\"/\"openshift-service-ca.crt\"" Sep 10 16:32:39.258260 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.258127 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-09da25\"/\"default-dockercfg-s6x8k\"" Sep 10 16:32:39.258353 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.258333 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-09da25\"/\"explicit-model-09da25-kserve-self-signed-certs\"" Sep 10 16:32:39.263742 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.263703 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h"] Sep 10 16:32:39.272896 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.272869 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:39.387495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.387458 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90022391-2387-47f2-959b-13209afae708-tls-certs\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.387678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.387576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlwd7\" (UniqueName: \"kubernetes.io/projected/90022391-2387-47f2-959b-13209afae708-kube-api-access-vlwd7\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.387678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.387629 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-home\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.387678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.387657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-model-cache\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.387799 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.387749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-dshm\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.387799 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.387784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-tmp-dir\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.488488 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.488449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwd7\" (UniqueName: \"kubernetes.io/projected/90022391-2387-47f2-959b-13209afae708-kube-api-access-vlwd7\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.488670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.488499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-home\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.488670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.488530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-model-cache\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.488670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.488589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-dshm\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.488670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.488620 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-tmp-dir\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.488670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.488650 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90022391-2387-47f2-959b-13209afae708-tls-certs\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.488946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.488925 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-model-cache\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.488992 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.488964 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-home\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.489092 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.489027 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-tmp-dir\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.490958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.490933 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-dshm\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.491204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.491186 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90022391-2387-47f2-959b-13209afae708-tls-certs\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.498711 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.498681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwd7\" (UniqueName: \"kubernetes.io/projected/90022391-2387-47f2-959b-13209afae708-kube-api-access-vlwd7\") pod \"explicit-model-09da25-kserve-f7bcc7747-m227h\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.564067 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.563917 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:39.839430 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.839403 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h"] Sep 10 16:32:39.842227 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:32:39.842197 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90022391_2387_47f2_959b_13209afae708.slice/crio-88a3afc1b72a68bd0d1b36f8dc258ef7c663ffe0cfc6a7b05837cb898b2562fd WatchSource:0}: Error finding container 88a3afc1b72a68bd0d1b36f8dc258ef7c663ffe0cfc6a7b05837cb898b2562fd: Status 404 returned error can't find the container with id 88a3afc1b72a68bd0d1b36f8dc258ef7c663ffe0cfc6a7b05837cb898b2562fd Sep 10 16:32:39.843048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:39.843011 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:32:40.629199 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:40.629159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" event={"ID":"90022391-2387-47f2-959b-13209afae708","Type":"ContainerStarted","Data":"d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8"} Sep 10 16:32:40.629643 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:40.629239 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" event={"ID":"90022391-2387-47f2-959b-13209afae708","Type":"ContainerStarted","Data":"88a3afc1b72a68bd0d1b36f8dc258ef7c663ffe0cfc6a7b05837cb898b2562fd"} Sep 10 16:32:40.649129 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:40.649075 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" podStartSLOduration=1.6490591380000001 podStartE2EDuration="1.649059138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:32:40.647541063 +0000 UTC m=+911.373859993" watchObservedRunningTime="2026-09-10 16:32:40.649059138 +0000 UTC m=+911.375378065" Sep 10 16:32:47.261049 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:47.261001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:49.564336 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:49.564292 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:49.564734 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:49.564387 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:49.576747 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:49.576721 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:49.687221 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:49.687190 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:32:52.844721 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:52.844686 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h"] Sep 10 16:32:52.872274 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:52.872239 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:52.962508 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:52.962472 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x"] Sep 10 16:32:52.966615 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:52.966590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" Sep 10 16:32:52.969277 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:52.969250 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:32:52.969417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:52.969364 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:32:52.972747 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:52.972724 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x"] Sep 10 16:32:53.035666 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.035621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-09da25-ksj6x\" (UID: \"c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" Sep 10 16:32:53.136985 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.136881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-09da25-ksj6x\" (UID: \"c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" Sep 10 16:32:53.137608 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.137581 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-09da25-ksj6x\" (UID: \"c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" Sep 10 16:32:53.279210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.279158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" Sep 10 16:32:53.514746 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.514712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:53.550811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.550787 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x"] Sep 10 16:32:53.553022 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:32:53.552990 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dfaf36_1bba_4e16_9228_b0a1f9b3a9df.slice/crio-06d5fcd0b468ca431a5e424302bb6715f738a4083bc33172351b03ae2c71086b WatchSource:0}: Error finding container 06d5fcd0b468ca431a5e424302bb6715f738a4083bc33172351b03ae2c71086b: Status 404 returned error can't find the container with id 06d5fcd0b468ca431a5e424302bb6715f738a4083bc33172351b03ae2c71086b Sep 10 16:32:53.615472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.615437 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:53.698835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.698798 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" event={"ID":"c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df","Type":"ContainerStarted","Data":"06d5fcd0b468ca431a5e424302bb6715f738a4083bc33172351b03ae2c71086b"} Sep 10 16:32:53.718017 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.717986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:53.869535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:53.869504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:54.497544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:54.497510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:54.705989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:54.705895 2562 generic.go:356] "Generic (PLEG): container finished" podID="c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df" containerID="e80d50fbd8e4b05a6dd348dd218780deb6f35226fd574302b2e9352f4c1fc01b" exitCode=0 Sep 10 16:32:54.706162 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:54.705983 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" event={"ID":"c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df","Type":"ContainerDied","Data":"e80d50fbd8e4b05a6dd348dd218780deb6f35226fd574302b2e9352f4c1fc01b"} Sep 10 16:32:55.152299 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:55.152264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:55.800243 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:55.800206 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:32:55.848394 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:55.848370 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" Sep 10 16:32:55.968732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:55.968637 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df-maas-api-service-ca\") pod \"c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df\" (UID: \"c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df\") " Sep 10 16:32:55.969018 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:55.968998 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df" (UID: "c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:32:56.069578 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:56.069539 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:32:56.719082 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:56.719030 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" Sep 10 16:32:56.719082 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:56.719075 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x" event={"ID":"c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df","Type":"ContainerDied","Data":"06d5fcd0b468ca431a5e424302bb6715f738a4083bc33172351b03ae2c71086b"} Sep 10 16:32:56.719532 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:32:56.719122 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d5fcd0b468ca431a5e424302bb6715f738a4083bc33172351b03ae2c71086b" Sep 10 16:33:00.237391 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:00.237355 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:33:03.492452 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.492416 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:03.515940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.515907 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z"] Sep 10 16:33:03.516239 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.516210 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" podUID="70539d7e-de24-4c4b-aece-2ebbb07554c2" containerName="payload-pre-processing" containerID="cri-o://18b6dc8eadda57ffd401e2cdd45bf1d563c250a02fd9101768ba4a6dbe275edb" gracePeriod=30 Sep 10 16:33:03.551651 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.551612 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:03.651808 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.651733 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x"] Sep 10 16:33:03.652751 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.652716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:03.653860 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.653837 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-09da25-ksj6x"] Sep 10 16:33:03.758324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.757385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:03.770987 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.770807 2562 generic.go:356] "Generic (PLEG): container finished" podID="70539d7e-de24-4c4b-aece-2ebbb07554c2" containerID="18b6dc8eadda57ffd401e2cdd45bf1d563c250a02fd9101768ba4a6dbe275edb" exitCode=0 Sep 10 16:33:03.771749 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.771726 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df" path="/var/lib/kubelet/pods/c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df/volumes" Sep 10 16:33:03.772156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.772128 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" event={"ID":"70539d7e-de24-4c4b-aece-2ebbb07554c2","Type":"ContainerDied","Data":"18b6dc8eadda57ffd401e2cdd45bf1d563c250a02fd9101768ba4a6dbe275edb"} Sep 10 16:33:03.794117 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.794091 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:33:03.848430 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.848395 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70539d7e-de24-4c4b-aece-2ebbb07554c2-plugins-config-volume\") pod \"70539d7e-de24-4c4b-aece-2ebbb07554c2\" (UID: \"70539d7e-de24-4c4b-aece-2ebbb07554c2\") " Sep 10 16:33:03.848600 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.848463 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lkqm\" (UniqueName: \"kubernetes.io/projected/70539d7e-de24-4c4b-aece-2ebbb07554c2-kube-api-access-8lkqm\") pod \"70539d7e-de24-4c4b-aece-2ebbb07554c2\" (UID: \"70539d7e-de24-4c4b-aece-2ebbb07554c2\") " Sep 10 16:33:03.848798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.848775 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70539d7e-de24-4c4b-aece-2ebbb07554c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70539d7e-de24-4c4b-aece-2ebbb07554c2" (UID: "70539d7e-de24-4c4b-aece-2ebbb07554c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:33:03.848902 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.848890 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70539d7e-de24-4c4b-aece-2ebbb07554c2-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:03.850671 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.850650 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70539d7e-de24-4c4b-aece-2ebbb07554c2-kube-api-access-8lkqm" (OuterVolumeSpecName: "kube-api-access-8lkqm") pod "70539d7e-de24-4c4b-aece-2ebbb07554c2" (UID: "70539d7e-de24-4c4b-aece-2ebbb07554c2"). InnerVolumeSpecName "kube-api-access-8lkqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:33:03.949765 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:03.949727 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lkqm\" (UniqueName: \"kubernetes.io/projected/70539d7e-de24-4c4b-aece-2ebbb07554c2-kube-api-access-8lkqm\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:04.009615 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:04.009522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:04.110194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:04.110159 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:04.673739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:04.673707 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:04.778566 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:04.778534 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" Sep 10 16:33:04.778566 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:04.778551 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z" event={"ID":"70539d7e-de24-4c4b-aece-2ebbb07554c2","Type":"ContainerDied","Data":"12fe86f5306505951b4a153311e25cf1c2f51333776e56f31837dae9d284a636"} Sep 10 16:33:04.778818 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:04.778626 2562 scope.go:117] "RemoveContainer" containerID="18b6dc8eadda57ffd401e2cdd45bf1d563c250a02fd9101768ba4a6dbe275edb" Sep 10 16:33:04.801005 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:04.800976 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z"] Sep 10 16:33:04.804537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:04.804512 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-09da25-86c5684754-bw74z"] Sep 10 16:33:05.499290 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:05.499258 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:05.757795 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:05.757716 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70539d7e-de24-4c4b-aece-2ebbb07554c2" path="/var/lib/kubelet/pods/70539d7e-de24-4c4b-aece-2ebbb07554c2/volumes" Sep 10 16:33:07.867153 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:07.867107 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" podUID="90022391-2387-47f2-959b-13209afae708" containerName="main" containerID="cri-o://d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8" gracePeriod=45 Sep 10 16:33:08.121402 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.121335 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:33:08.196436 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196400 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90022391-2387-47f2-959b-13209afae708-tls-certs\") pod \"90022391-2387-47f2-959b-13209afae708\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " Sep 10 16:33:08.196635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196504 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-model-cache\") pod \"90022391-2387-47f2-959b-13209afae708\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " Sep 10 16:33:08.196635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196560 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-home\") pod \"90022391-2387-47f2-959b-13209afae708\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " Sep 10 16:33:08.196635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196604 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlwd7\" (UniqueName: \"kubernetes.io/projected/90022391-2387-47f2-959b-13209afae708-kube-api-access-vlwd7\") pod \"90022391-2387-47f2-959b-13209afae708\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " Sep 10 16:33:08.196799 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196641 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-tmp-dir\") pod \"90022391-2387-47f2-959b-13209afae708\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " Sep 10 16:33:08.196799 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196694 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-dshm\") pod \"90022391-2387-47f2-959b-13209afae708\" (UID: \"90022391-2387-47f2-959b-13209afae708\") " Sep 10 16:33:08.196895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196802 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-model-cache" (OuterVolumeSpecName: "model-cache") pod "90022391-2387-47f2-959b-13209afae708" (UID: "90022391-2387-47f2-959b-13209afae708"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:08.196895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196818 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-home" (OuterVolumeSpecName: "home") pod "90022391-2387-47f2-959b-13209afae708" (UID: "90022391-2387-47f2-959b-13209afae708"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:08.196989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.196958 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90022391-2387-47f2-959b-13209afae708" (UID: "90022391-2387-47f2-959b-13209afae708"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:08.197120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.197103 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-model-cache\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:08.197189 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.197124 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-home\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:08.197189 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.197153 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-tmp-dir\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:08.198785 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.198761 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-dshm" (OuterVolumeSpecName: "dshm") pod "90022391-2387-47f2-959b-13209afae708" (UID: "90022391-2387-47f2-959b-13209afae708"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:08.198861 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.198808 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90022391-2387-47f2-959b-13209afae708-kube-api-access-vlwd7" (OuterVolumeSpecName: "kube-api-access-vlwd7") pod "90022391-2387-47f2-959b-13209afae708" (UID: "90022391-2387-47f2-959b-13209afae708"). InnerVolumeSpecName "kube-api-access-vlwd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:33:08.198861 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.198827 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90022391-2387-47f2-959b-13209afae708-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90022391-2387-47f2-959b-13209afae708" (UID: "90022391-2387-47f2-959b-13209afae708"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:33:08.298265 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.298230 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlwd7\" (UniqueName: \"kubernetes.io/projected/90022391-2387-47f2-959b-13209afae708-kube-api-access-vlwd7\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:08.298265 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.298259 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90022391-2387-47f2-959b-13209afae708-dshm\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:08.298473 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.298268 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90022391-2387-47f2-959b-13209afae708-tls-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:08.802194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.802160 2562 generic.go:356] "Generic (PLEG): container finished" podID="90022391-2387-47f2-959b-13209afae708" containerID="d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8" exitCode=0 Sep 10 16:33:08.802374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.802203 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" Sep 10 16:33:08.802374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.802248 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" event={"ID":"90022391-2387-47f2-959b-13209afae708","Type":"ContainerDied","Data":"d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8"} Sep 10 16:33:08.802374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.802294 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h" event={"ID":"90022391-2387-47f2-959b-13209afae708","Type":"ContainerDied","Data":"88a3afc1b72a68bd0d1b36f8dc258ef7c663ffe0cfc6a7b05837cb898b2562fd"} Sep 10 16:33:08.802374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.802316 2562 scope.go:117] "RemoveContainer" containerID="d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8" Sep 10 16:33:08.811751 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.811730 2562 scope.go:117] "RemoveContainer" containerID="d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8" Sep 10 16:33:08.812095 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:08.812073 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8\": container with ID starting with d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8 not found: ID does not exist" containerID="d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8" Sep 10 16:33:08.812161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.812105 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8"} err="failed to get container status \"d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8\": rpc error: code = NotFound desc = could not find container \"d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8\": container with ID starting with d2b2016c5bd26f5b919627c9a4aab6d50f727ee776b53adfe77e62a41b6a1ff8 not found: ID does not exist" Sep 10 16:33:08.822951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.822924 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h"] Sep 10 16:33:08.828629 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:08.828601 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-09da25/explicit-model-09da25-kserve-f7bcc7747-m227h"] Sep 10 16:33:09.538683 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:09.538644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:09.758198 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:09.758165 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90022391-2387-47f2-959b-13209afae708" path="/var/lib/kubelet/pods/90022391-2387-47f2-959b-13209afae708/volumes" Sep 10 16:33:10.398472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:10.398439 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:10.516981 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:10.516947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:10.664549 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:10.664464 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf"] Sep 10 16:33:10.664956 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:10.664671 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" containerID="cri-o://b05c067714c2b19a3be2136457b38fc193e07375335974720c3997dc8d5478ee" gracePeriod=30 Sep 10 16:33:10.680615 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:10.680585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:11.126197 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.126165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:11.389564 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.389482 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92"] Sep 10 16:33:11.390026 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390006 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df" containerName="revoke-keys" Sep 10 16:33:11.390125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390029 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df" containerName="revoke-keys" Sep 10 16:33:11.390125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390075 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90022391-2387-47f2-959b-13209afae708" containerName="main" Sep 10 16:33:11.390125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390085 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90022391-2387-47f2-959b-13209afae708" containerName="main" Sep 10 16:33:11.390125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390115 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70539d7e-de24-4c4b-aece-2ebbb07554c2" containerName="payload-pre-processing" Sep 10 16:33:11.390125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390125 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70539d7e-de24-4c4b-aece-2ebbb07554c2" containerName="payload-pre-processing" Sep 10 16:33:11.390292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390210 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="90022391-2387-47f2-959b-13209afae708" containerName="main" Sep 10 16:33:11.390292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390220 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="70539d7e-de24-4c4b-aece-2ebbb07554c2" containerName="payload-pre-processing" Sep 10 16:33:11.390292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.390232 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9dfaf36-1bba-4e16-9228-b0a1f9b3a9df" containerName="revoke-keys" Sep 10 16:33:11.394840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.394816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.397831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.397807 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-496136-openshift-default-dockercfg-4rvrt\"" Sep 10 16:33:11.405622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.405596 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92"] Sep 10 16:33:11.408268 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.408245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:11.530198 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-data\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.530198 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-certs\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.530538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530362 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-credential-socket\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.530538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530402 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-token\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.530538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530427 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istiod-ca-cert\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.530538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530456 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnz9l\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-kube-api-access-dnz9l\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.530538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-podinfo\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.530804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530548 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-envoy\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.530804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.530630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-socket\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.550952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.550921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:11.631772 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.631730 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-credential-socket\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.631974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.631792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-token\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.631974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.631821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istiod-ca-cert\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.631974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.631860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnz9l\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-kube-api-access-dnz9l\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.631974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.631886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-podinfo\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.631974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.631914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-envoy\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.631974 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.631964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-socket\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.632352 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.631998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-data\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.632352 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.632049 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-certs\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.632352 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.632225 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-credential-socket\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.632522 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.632411 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-certs\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.632522 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.632502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-socket\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.632955 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.632652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-data\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.633248 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.633224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istiod-ca-cert\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.635304 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.635189 2562 patch_prober.go:28] interesting pod/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:33:11.635304 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.635240 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:33:11.635997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.635590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-envoy\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.635997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.635617 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-podinfo\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.642983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.642884 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-token\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.643582 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.643558 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnz9l\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-kube-api-access-dnz9l\") pod \"e2e-noten-496136-openshift-default-b54b94466-ltb92\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.706316 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.706282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:11.985898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.985871 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92"] Sep 10 16:33:11.987941 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:33:11.987916 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3244dc30_00b7_4b3e_92d4_fdbbf9060b6d.slice/crio-2cfed4cd31148ab60604c0d16c3a91dc76cdd45bf82e9ce2f3c66d5f8badc4c7 WatchSource:0}: Error finding container 2cfed4cd31148ab60604c0d16c3a91dc76cdd45bf82e9ce2f3c66d5f8badc4c7: Status 404 returned error can't find the container with id 2cfed4cd31148ab60604c0d16c3a91dc76cdd45bf82e9ce2f3c66d5f8badc4c7 Sep 10 16:33:11.990367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.990324 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:33:11.990468 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.990421 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:33:11.990510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:11.990471 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:33:12.826022 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:12.825987 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" event={"ID":"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d","Type":"ContainerStarted","Data":"639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e"} Sep 10 16:33:12.826438 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:12.826165 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" event={"ID":"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d","Type":"ContainerStarted","Data":"2cfed4cd31148ab60604c0d16c3a91dc76cdd45bf82e9ce2f3c66d5f8badc4c7"} Sep 10 16:33:12.850136 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:12.850081 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" podStartSLOduration=1.850063461 podStartE2EDuration="1.850063461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:33:12.84700482 +0000 UTC m=+943.573323797" watchObservedRunningTime="2026-09-10 16:33:12.850063461 +0000 UTC m=+943.576382390" Sep 10 16:33:13.115398 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:13.115270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:13.215107 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:13.215071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:13.317221 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:13.317187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:13.707408 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:13.707369 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:13.712668 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:13.712639 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:13.838204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:13.838168 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:13.839747 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:13.839721 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:14.639507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:14.639474 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:33:15.825998 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:15.825915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:15.855892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:15.855856 2562 generic.go:356] "Generic (PLEG): container finished" podID="37cb139d-fdb3-4eea-a171-7041fd052503" containerID="b05c067714c2b19a3be2136457b38fc193e07375335974720c3997dc8d5478ee" exitCode=0 Sep 10 16:33:15.856111 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:15.856060 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" event={"ID":"37cb139d-fdb3-4eea-a171-7041fd052503","Type":"ContainerDied","Data":"b05c067714c2b19a3be2136457b38fc193e07375335974720c3997dc8d5478ee"} Sep 10 16:33:15.856193 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:15.856163 2562 scope.go:117] "RemoveContainer" containerID="23812e6cd5632a22890834b69cc394925aef9b1ebc084374b5b400063eb84175" Sep 10 16:33:15.889600 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:15.889562 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:15.926782 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:15.926706 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:33:15.968061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:15.968006 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:16.029019 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.028954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:16.081142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081109 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-credential-socket\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081163 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-socket\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081182 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-certs\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081212 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-istio-token\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081244 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-envoy\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081278 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-data\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081308 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs5lb\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-kube-api-access-vs5lb\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081379 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cb139d-fdb3-4eea-a171-7041fd052503-istiod-ca-cert\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081409 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cb139d-fdb3-4eea-a171-7041fd052503-istio-podinfo\") pod \"37cb139d-fdb3-4eea-a171-7041fd052503\" (UID: \"37cb139d-fdb3-4eea-a171-7041fd052503\") " Sep 10 16:33:16.081635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081454 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:16.081635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081469 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:16.081839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081690 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-data" (OuterVolumeSpecName: "istio-data") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:16.081839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081793 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.081839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081815 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.081980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081856 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.081980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.081795 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cb139d-fdb3-4eea-a171-7041fd052503-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:33:16.082120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.082093 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:16.083766 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.083734 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37cb139d-fdb3-4eea-a171-7041fd052503-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:33:16.084106 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.084089 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-istio-token" (OuterVolumeSpecName: "istio-token") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:33:16.084264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.084237 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:16.084324 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.084245 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-kube-api-access-vs5lb" (OuterVolumeSpecName: "kube-api-access-vs5lb") pod "37cb139d-fdb3-4eea-a171-7041fd052503" (UID: "37cb139d-fdb3-4eea-a171-7041fd052503"). InnerVolumeSpecName "kube-api-access-vs5lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:33:16.183091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.182976 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.183091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.183005 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.183091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.183014 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cb139d-fdb3-4eea-a171-7041fd052503-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.183091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.183022 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs5lb\" (UniqueName: \"kubernetes.io/projected/37cb139d-fdb3-4eea-a171-7041fd052503-kube-api-access-vs5lb\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.183091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.183030 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cb139d-fdb3-4eea-a171-7041fd052503-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.183091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.183074 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cb139d-fdb3-4eea-a171-7041fd052503-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:16.862456 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.862429 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" Sep 10 16:33:16.862858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.862472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf" event={"ID":"37cb139d-fdb3-4eea-a171-7041fd052503","Type":"ContainerDied","Data":"0289fd72e7d1c8d4545b7402c45e22cbc3582818771c638d4fe41eb2bad8d698"} Sep 10 16:33:16.862858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.862533 2562 scope.go:117] "RemoveContainer" containerID="b05c067714c2b19a3be2136457b38fc193e07375335974720c3997dc8d5478ee" Sep 10 16:33:16.884986 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.884953 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf"] Sep 10 16:33:16.888765 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:16.888736 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-09da25-openshift-default-5b54b85b88-pxzsf"] Sep 10 16:33:17.756977 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:17.756933 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" path="/var/lib/kubelet/pods/37cb139d-fdb3-4eea-a171-7041fd052503/volumes" Sep 10 16:33:26.490022 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:26.489981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:27.260309 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:27.260273 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:29.537102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:29.537062 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:33:33.732848 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:33.732737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:33.793307 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:33.793259 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:33.870300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:33.870267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:33.970567 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:33.970536 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:34.025889 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:34.025854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:34.702276 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:34.702235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:34.801679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:34.801643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:34.897997 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:34.897966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:36.317015 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:36.316940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:42.752174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:42.752143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:42.822534 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:42.822492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:43.145687 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.145602 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:33:43.413297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.413215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:43.444835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.444799 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84"] Sep 10 16:33:43.445483 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.445462 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" Sep 10 16:33:43.445539 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.445491 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" Sep 10 16:33:43.445539 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.445503 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" Sep 10 16:33:43.445539 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.445513 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" Sep 10 16:33:43.445639 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.445629 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" Sep 10 16:33:43.445674 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.445642 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cb139d-fdb3-4eea-a171-7041fd052503" containerName="istio-proxy" Sep 10 16:33:43.450741 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.450714 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.454942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.454909 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-496136\"/\"kube-root-ca.crt\"" Sep 10 16:33:43.454942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.454927 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-496136\"/\"openshift-service-ca.crt\"" Sep 10 16:33:43.455184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.455145 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-496136\"/\"default-dockercfg-8brsd\"" Sep 10 16:33:43.455269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.455251 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-496136\"/\"noten-model-496136-kserve-self-signed-certs\"" Sep 10 16:33:43.457641 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.457618 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84"] Sep 10 16:33:43.476531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.476501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:43.566127 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.566086 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-dshm\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.566310 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.566164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-model-cache\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.566310 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.566196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4m8\" (UniqueName: \"kubernetes.io/projected/e3657142-31a5-44ae-8705-092bf19f5ec9-kube-api-access-km4m8\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.566310 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.566222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.566432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.566316 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-home\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.566432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.566354 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-tmp-dir\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667480 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-home\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667480 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667444 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-tmp-dir\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667493 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-dshm\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-model-cache\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4m8\" (UniqueName: \"kubernetes.io/projected/e3657142-31a5-44ae-8705-092bf19f5ec9-kube-api-access-km4m8\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667621 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667832 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667786 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-home\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-tmp-dir\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.667873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.667862 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-model-cache\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.669855 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.669835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-dshm\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.670198 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.670181 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.675425 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.675401 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4m8\" (UniqueName: \"kubernetes.io/projected/e3657142-31a5-44ae-8705-092bf19f5ec9-kube-api-access-km4m8\") pod \"noten-model-496136-kserve-5446b447d7-jhk84\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:43.762813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:43.762776 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:44.036449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:44.036417 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84"] Sep 10 16:33:44.038150 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:33:44.038125 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3657142_31a5_44ae_8705_092bf19f5ec9.slice/crio-f7a7b015fec068d1ff0a505d368a63f1225af8353bfd6cd1fed0c5592a160471 WatchSource:0}: Error finding container f7a7b015fec068d1ff0a505d368a63f1225af8353bfd6cd1fed0c5592a160471: Status 404 returned error can't find the container with id f7a7b015fec068d1ff0a505d368a63f1225af8353bfd6cd1fed0c5592a160471 Sep 10 16:33:45.014292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:45.014253 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" event={"ID":"e3657142-31a5-44ae-8705-092bf19f5ec9","Type":"ContainerStarted","Data":"17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a"} Sep 10 16:33:45.014701 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:45.014312 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" event={"ID":"e3657142-31a5-44ae-8705-092bf19f5ec9","Type":"ContainerStarted","Data":"f7a7b015fec068d1ff0a505d368a63f1225af8353bfd6cd1fed0c5592a160471"} Sep 10 16:33:45.032715 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:45.032663 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" podStartSLOduration=2.03264513 podStartE2EDuration="2.03264513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:33:45.031198334 +0000 UTC m=+975.757517264" watchObservedRunningTime="2026-09-10 16:33:45.03264513 +0000 UTC m=+975.758964059" Sep 10 16:33:45.876257 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:45.876218 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84"] Sep 10 16:33:45.890292 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:45.890258 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:45.988337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:45.988306 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92"] Sep 10 16:33:45.988524 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:45.988502 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" podUID="3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" containerName="istio-proxy" containerID="cri-o://639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e" gracePeriod=30 Sep 10 16:33:46.003910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:46.003866 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:46.193075 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:46.192963 2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-496136/noten-model-496136-kserve-self-signed-certs: secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:46.193441 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:46.193079 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs podName:e3657142-31a5-44ae-8705-092bf19f5ec9 nodeName:}" failed. No retries permitted until 2026-09-10 16:33:46.693054514 +0000 UTC m=+977.419373440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs") pod "noten-model-496136-kserve-5446b447d7-jhk84" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9") : secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:46.218705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:46.218671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:46.328937 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:46.328903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:46.699246 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:46.699212 2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-496136/noten-model-496136-kserve-self-signed-certs: secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:46.699559 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:46.699542 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs podName:e3657142-31a5-44ae-8705-092bf19f5ec9 nodeName:}" failed. No retries permitted until 2026-09-10 16:33:47.699508863 +0000 UTC m=+978.425827791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs") pod "noten-model-496136-kserve-5446b447d7-jhk84" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9") : secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:47.646700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:47.646659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:47.708963 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:47.708924 2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-496136/noten-model-496136-kserve-self-signed-certs: secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:47.709172 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:47.709010 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs podName:e3657142-31a5-44ae-8705-092bf19f5ec9 nodeName:}" failed. No retries permitted until 2026-09-10 16:33:49.708994226 +0000 UTC m=+980.435313139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs") pod "noten-model-496136-kserve-5446b447d7-jhk84" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9") : secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:48.755684 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:48.755645 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:48.829413 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:48.829376 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2"] Sep 10 16:33:48.833090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:48.833063 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" Sep 10 16:33:48.837787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:48.837349 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:33:48.837787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:48.837631 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:33:48.844164 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:48.843516 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2"] Sep 10 16:33:49.024722 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:49.024599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f7754-c17b-44ab-89b7-1ece7f8d16ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2\" (UID: \"b68f7754-c17b-44ab-89b7-1ece7f8d16ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" Sep 10 16:33:49.125912 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:49.125877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f7754-c17b-44ab-89b7-1ece7f8d16ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2\" (UID: \"b68f7754-c17b-44ab-89b7-1ece7f8d16ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" Sep 10 16:33:49.126621 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:49.126597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f7754-c17b-44ab-89b7-1ece7f8d16ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2\" (UID: \"b68f7754-c17b-44ab-89b7-1ece7f8d16ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" Sep 10 16:33:49.140148 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:49.140111 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:49.156123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:49.156090 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" Sep 10 16:33:49.436093 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:49.436068 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2"] Sep 10 16:33:49.438819 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:33:49.438789 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68f7754_c17b_44ab_89b7_1ece7f8d16ea.slice/crio-b80a2d8b00c30b012c49e761f3db7721d8b260aca19758cac949d214a4c53552 WatchSource:0}: Error finding container b80a2d8b00c30b012c49e761f3db7721d8b260aca19758cac949d214a4c53552: Status 404 returned error can't find the container with id b80a2d8b00c30b012c49e761f3db7721d8b260aca19758cac949d214a4c53552 Sep 10 16:33:49.731482 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:49.731387 2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-496136/noten-model-496136-kserve-self-signed-certs: secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:49.731482 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:49.731468 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs podName:e3657142-31a5-44ae-8705-092bf19f5ec9 nodeName:}" failed. No retries permitted until 2026-09-10 16:33:53.731449453 +0000 UTC m=+984.457768374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs") pod "noten-model-496136-kserve-5446b447d7-jhk84" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9") : secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:49.860108 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:49.860074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:50.044320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:50.044278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" event={"ID":"b68f7754-c17b-44ab-89b7-1ece7f8d16ea","Type":"ContainerStarted","Data":"881a0ed430a0991a685505343e1a57e511dacb5b1171335fa2dd37e342c88814"} Sep 10 16:33:50.044486 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:50.044352 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" event={"ID":"b68f7754-c17b-44ab-89b7-1ece7f8d16ea","Type":"ContainerStarted","Data":"b80a2d8b00c30b012c49e761f3db7721d8b260aca19758cac949d214a4c53552"} Sep 10 16:33:50.061403 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:50.061345 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" podStartSLOduration=2.061324679 podStartE2EDuration="2.061324679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:33:50.060772703 +0000 UTC m=+980.787091638" watchObservedRunningTime="2026-09-10 16:33:50.061324679 +0000 UTC m=+980.787643654" Sep 10 16:33:50.465330 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:50.465220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:50.992546 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:50.992500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:51.051199 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.051166 2562 generic.go:356] "Generic (PLEG): container finished" podID="b68f7754-c17b-44ab-89b7-1ece7f8d16ea" containerID="881a0ed430a0991a685505343e1a57e511dacb5b1171335fa2dd37e342c88814" exitCode=0 Sep 10 16:33:51.051383 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.051254 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" event={"ID":"b68f7754-c17b-44ab-89b7-1ece7f8d16ea","Type":"ContainerDied","Data":"881a0ed430a0991a685505343e1a57e511dacb5b1171335fa2dd37e342c88814"} Sep 10 16:33:51.108749 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.108720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:51.248826 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.248801 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:51.258560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.258533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:51.346909 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.346868 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-certs\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.346920 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-data\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.346957 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-token\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.346987 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-socket\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347052 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-envoy\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347097 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-credential-socket\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347447 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347165 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnz9l\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-kube-api-access-dnz9l\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347447 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347168 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:51.347447 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347216 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istiod-ca-cert\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347447 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347254 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-podinfo\") pod \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\" (UID: \"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d\") " Sep 10 16:33:51.347635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347249 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:51.347635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347378 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:51.347635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347433 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:51.347635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:33:51.347856 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347657 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.347856 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347674 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.347856 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347685 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.347856 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.347692 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.349293 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.349264 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:33:51.349446 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.349425 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-kube-api-access-dnz9l" (OuterVolumeSpecName: "kube-api-access-dnz9l") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "kube-api-access-dnz9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:33:51.349635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.349612 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:33:51.349729 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.349655 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" (UID: "3244dc30-00b7-4b3e-92d4-fdbbf9060b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:33:51.358983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.358961 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:51.448367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.448323 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.448367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.448356 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.448367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.448367 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnz9l\" (UniqueName: \"kubernetes.io/projected/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-kube-api-access-dnz9l\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.448367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.448378 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.448639 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.448387 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:51.463275 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.463240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:51.521457 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.521381 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr"] Sep 10 16:33:51.521832 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.521817 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" containerName="istio-proxy" Sep 10 16:33:51.521882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.521834 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" containerName="istio-proxy" Sep 10 16:33:51.521916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.521889 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" containerName="istio-proxy" Sep 10 16:33:51.525311 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.525293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.528143 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.528123 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e6e77069-openshift-default-dockercfg-jj4dh\"" Sep 10 16:33:51.537641 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.537604 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr"] Sep 10 16:33:51.549780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.549751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istiod-ca-cert\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.549908 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.549794 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-envoy\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.549908 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.549822 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-credential-socket\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.550009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.549929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-podinfo\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.550009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.549977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-data\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.550168 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.550151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-token\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.550209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.550187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-socket\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.550277 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.550261 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-certs\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.550312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.550289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v94m\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-kube-api-access-7v94m\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.560017 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.559988 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:51.651572 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.651530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-envoy\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.651572 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.651578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-credential-socket\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.651841 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.651721 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-podinfo\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.651841 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.651779 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-data\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.651946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.651931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-token\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652001 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.651966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-socket\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652001 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.651968 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-credential-socket\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652157 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.652073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-certs\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652157 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.652106 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v94m\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-kube-api-access-7v94m\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652157 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.652143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-data\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.652160 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istiod-ca-cert\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652378 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.652318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-socket\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652430 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.652405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-certs\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.652760 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.652742 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istiod-ca-cert\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.654066 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.654016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-envoy\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.654264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.654248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-podinfo\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.660369 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.660339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-token\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.660472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.660446 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v94m\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-kube-api-access-7v94m\") pod \"e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:51.729664 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.729630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:51.836659 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:51.836562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:52.057843 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.057805 2562 generic.go:356] "Generic (PLEG): container finished" podID="3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" containerID="639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e" exitCode=0 Sep 10 16:33:52.058332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.057855 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" Sep 10 16:33:52.058332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.057897 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" event={"ID":"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d","Type":"ContainerDied","Data":"639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e"} Sep 10 16:33:52.058332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.057947 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92" event={"ID":"3244dc30-00b7-4b3e-92d4-fdbbf9060b6d","Type":"ContainerDied","Data":"2cfed4cd31148ab60604c0d16c3a91dc76cdd45bf82e9ce2f3c66d5f8badc4c7"} Sep 10 16:33:52.058332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.057971 2562 scope.go:117] "RemoveContainer" containerID="639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e" Sep 10 16:33:52.069529 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.069505 2562 scope.go:117] "RemoveContainer" containerID="639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e" Sep 10 16:33:52.069850 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:52.069830 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e\": container with ID starting with 639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e not found: ID does not exist" containerID="639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e" Sep 10 16:33:52.069939 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.069864 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e"} err="failed to get container status \"639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e\": rpc error: code = NotFound desc = could not find container \"639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e\": container with ID starting with 639025e6d10e1500fc0f45dedec5b7ebae92b6410d9d95cf909f41954896011e not found: ID does not exist" Sep 10 16:33:52.076666 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.076634 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92"] Sep 10 16:33:52.082322 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.082294 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-496136-openshift-default-b54b94466-ltb92"] Sep 10 16:33:52.111657 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.111629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr"] Sep 10 16:33:52.113549 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:33:52.113515 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ae9684_7dd5_42fd_a345_b4cb23d150a6.slice/crio-d09378be3c11477f7b1f92ecfc64db889b98124dad6ef7313f7b2261e6b857d8 WatchSource:0}: Error finding container d09378be3c11477f7b1f92ecfc64db889b98124dad6ef7313f7b2261e6b857d8: Status 404 returned error can't find the container with id d09378be3c11477f7b1f92ecfc64db889b98124dad6ef7313f7b2261e6b857d8 Sep 10 16:33:52.116728 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.116686 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:33:52.116932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.116783 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:33:52.116932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.116830 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:33:52.186825 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.186801 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" Sep 10 16:33:52.258024 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.257984 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f7754-c17b-44ab-89b7-1ece7f8d16ea-maas-api-service-ca\") pod \"b68f7754-c17b-44ab-89b7-1ece7f8d16ea\" (UID: \"b68f7754-c17b-44ab-89b7-1ece7f8d16ea\") " Sep 10 16:33:52.258371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.258345 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68f7754-c17b-44ab-89b7-1ece7f8d16ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b68f7754-c17b-44ab-89b7-1ece7f8d16ea" (UID: "b68f7754-c17b-44ab-89b7-1ece7f8d16ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:33:52.258507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:52.258493 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f7754-c17b-44ab-89b7-1ece7f8d16ea-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:33:53.065280 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.065236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" event={"ID":"a5ae9684-7dd5-42fd-a345-b4cb23d150a6","Type":"ContainerStarted","Data":"74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128"} Sep 10 16:33:53.065710 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.065338 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" event={"ID":"a5ae9684-7dd5-42fd-a345-b4cb23d150a6","Type":"ContainerStarted","Data":"d09378be3c11477f7b1f92ecfc64db889b98124dad6ef7313f7b2261e6b857d8"} Sep 10 16:33:53.066650 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.066624 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" Sep 10 16:33:53.066774 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.066628 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2" event={"ID":"b68f7754-c17b-44ab-89b7-1ece7f8d16ea","Type":"ContainerDied","Data":"b80a2d8b00c30b012c49e761f3db7721d8b260aca19758cac949d214a4c53552"} Sep 10 16:33:53.066774 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.066748 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80a2d8b00c30b012c49e761f3db7721d8b260aca19758cac949d214a4c53552" Sep 10 16:33:53.088713 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.088633 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" podStartSLOduration=2.088611488 podStartE2EDuration="2.088611488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:33:53.088239998 +0000 UTC m=+983.814558943" watchObservedRunningTime="2026-09-10 16:33:53.088611488 +0000 UTC m=+983.814930419" Sep 10 16:33:53.758526 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.758479 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3244dc30-00b7-4b3e-92d4-fdbbf9060b6d" path="/var/lib/kubelet/pods/3244dc30-00b7-4b3e-92d4-fdbbf9060b6d/volumes" Sep 10 16:33:53.763120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.763095 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:33:53.774787 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:53.774755 2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-496136/noten-model-496136-kserve-self-signed-certs: secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:53.774892 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:33:53.774858 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs podName:e3657142-31a5-44ae-8705-092bf19f5ec9 nodeName:}" failed. No retries permitted until 2026-09-10 16:34:01.774834696 +0000 UTC m=+992.501153609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs") pod "noten-model-496136-kserve-5446b447d7-jhk84" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9") : secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:33:53.836965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.836931 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:53.841994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:53.841966 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:54.073720 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:54.073623 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:54.074569 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:54.074542 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:33:54.767735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:54.767696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:55.025495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:55.025394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:55.123268 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:55.123221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:55.569182 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:55.569140 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:56.283705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.283662 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:56.380266 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.380226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:56.481311 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.481279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:56.731356 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.731253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:56.824471 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.824435 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:56.830954 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.830920 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9"] Sep 10 16:33:56.832030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.831594 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b68f7754-c17b-44ab-89b7-1ece7f8d16ea" containerName="revoke-keys" Sep 10 16:33:56.832030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.831615 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b68f7754-c17b-44ab-89b7-1ece7f8d16ea" containerName="revoke-keys" Sep 10 16:33:56.832030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.831708 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b68f7754-c17b-44ab-89b7-1ece7f8d16ea" containerName="revoke-keys" Sep 10 16:33:56.839676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.839645 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:56.843506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.843313 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e6e77069\"" Sep 10 16:33:56.844180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.844149 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e6e77069-dockercfg-r6nv6\"" Sep 10 16:33:56.845372 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.845347 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9"] Sep 10 16:33:56.867875 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.867844 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx"] Sep 10 16:33:56.871773 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.871750 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:56.883631 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.883606 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx"] Sep 10 16:33:56.908376 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.908251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l54jh\" (UniqueName: \"kubernetes.io/projected/af090985-439e-4c1b-aee6-04af76d70104-kube-api-access-l54jh\") pod \"payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx\" (UID: \"af090985-439e-4c1b-aee6-04af76d70104\") " pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:56.908626 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.908380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba14a01-412a-4116-b65e-98cee7191cbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9\" (UID: \"fba14a01-412a-4116-b65e-98cee7191cbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:56.908626 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.908491 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af090985-439e-4c1b-aee6-04af76d70104-plugins-config-volume\") pod \"payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx\" (UID: \"af090985-439e-4c1b-aee6-04af76d70104\") " pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:56.908626 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:56.908612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwxw\" (UniqueName: \"kubernetes.io/projected/fba14a01-412a-4116-b65e-98cee7191cbc-kube-api-access-5dwxw\") pod \"payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9\" (UID: \"fba14a01-412a-4116-b65e-98cee7191cbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:57.009616 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.009578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwxw\" (UniqueName: \"kubernetes.io/projected/fba14a01-412a-4116-b65e-98cee7191cbc-kube-api-access-5dwxw\") pod \"payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9\" (UID: \"fba14a01-412a-4116-b65e-98cee7191cbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:57.009774 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.009653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l54jh\" (UniqueName: \"kubernetes.io/projected/af090985-439e-4c1b-aee6-04af76d70104-kube-api-access-l54jh\") pod \"payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx\" (UID: \"af090985-439e-4c1b-aee6-04af76d70104\") " pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:57.009774 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.009672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba14a01-412a-4116-b65e-98cee7191cbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9\" (UID: \"fba14a01-412a-4116-b65e-98cee7191cbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:57.009774 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.009718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af090985-439e-4c1b-aee6-04af76d70104-plugins-config-volume\") pod \"payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx\" (UID: \"af090985-439e-4c1b-aee6-04af76d70104\") " pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:57.010482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.010461 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba14a01-412a-4116-b65e-98cee7191cbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9\" (UID: \"fba14a01-412a-4116-b65e-98cee7191cbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:57.010538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.010480 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af090985-439e-4c1b-aee6-04af76d70104-plugins-config-volume\") pod \"payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx\" (UID: \"af090985-439e-4c1b-aee6-04af76d70104\") " pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:57.017843 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.017810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l54jh\" (UniqueName: \"kubernetes.io/projected/af090985-439e-4c1b-aee6-04af76d70104-kube-api-access-l54jh\") pod \"payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx\" (UID: \"af090985-439e-4c1b-aee6-04af76d70104\") " pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:57.017989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.017857 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwxw\" (UniqueName: \"kubernetes.io/projected/fba14a01-412a-4116-b65e-98cee7191cbc-kube-api-access-5dwxw\") pod \"payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9\" (UID: \"fba14a01-412a-4116-b65e-98cee7191cbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:57.151193 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.151154 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:57.182550 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.182516 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:57.426761 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.426729 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9"] Sep 10 16:33:57.428772 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:33:57.428746 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba14a01_412a_4116_b65e_98cee7191cbc.slice/crio-229d4d0633e6cfc614f0efcb43d900b1fea10b1c65ffcac54afd54c0fe26d47e WatchSource:0}: Error finding container 229d4d0633e6cfc614f0efcb43d900b1fea10b1c65ffcac54afd54c0fe26d47e: Status 404 returned error can't find the container with id 229d4d0633e6cfc614f0efcb43d900b1fea10b1c65ffcac54afd54c0fe26d47e Sep 10 16:33:57.468260 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:57.468233 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx"] Sep 10 16:33:57.471551 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:33:57.471527 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf090985_439e_4c1b_aee6_04af76d70104.slice/crio-cc31a9621274e5a8011fb5da06f1338b43d1a8cefa2fcbb9989b6c7de170738a WatchSource:0}: Error finding container cc31a9621274e5a8011fb5da06f1338b43d1a8cefa2fcbb9989b6c7de170738a: Status 404 returned error can't find the container with id cc31a9621274e5a8011fb5da06f1338b43d1a8cefa2fcbb9989b6c7de170738a Sep 10 16:33:58.098776 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.098737 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" event={"ID":"af090985-439e-4c1b-aee6-04af76d70104","Type":"ContainerStarted","Data":"3bd9f3fc47ccfe79dff7e8137b5847874682b3cc0d34b4dfc615c61950de9764"} Sep 10 16:33:58.099009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.098797 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" event={"ID":"af090985-439e-4c1b-aee6-04af76d70104","Type":"ContainerStarted","Data":"cc31a9621274e5a8011fb5da06f1338b43d1a8cefa2fcbb9989b6c7de170738a"} Sep 10 16:33:58.099009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.098898 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:33:58.100516 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.100486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" event={"ID":"fba14a01-412a-4116-b65e-98cee7191cbc","Type":"ContainerStarted","Data":"e333dd9ff70339e17fcc1cc11813e9d3561e36e54d939ea36666b61cb912f095"} Sep 10 16:33:58.100640 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.100532 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" event={"ID":"fba14a01-412a-4116-b65e-98cee7191cbc","Type":"ContainerStarted","Data":"229d4d0633e6cfc614f0efcb43d900b1fea10b1c65ffcac54afd54c0fe26d47e"} Sep 10 16:33:58.100640 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.100578 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:33:58.119406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.119344 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" podStartSLOduration=2.119327269 podStartE2EDuration="2.119327269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:33:58.118458948 +0000 UTC m=+988.844777880" watchObservedRunningTime="2026-09-10 16:33:58.119327269 +0000 UTC m=+988.845646198" Sep 10 16:33:58.135630 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.135570 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" podStartSLOduration=2.135547667 podStartE2EDuration="2.135547667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:33:58.135168 +0000 UTC m=+988.861486930" watchObservedRunningTime="2026-09-10 16:33:58.135547667 +0000 UTC m=+988.861866651" Sep 10 16:33:58.967999 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:58.967969 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:59.053507 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:59.053464 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:33:59.592420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:59.592389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:59.670229 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:59.670186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:59.736449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:59.736418 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:33:59.784349 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:59.784314 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2"] Sep 10 16:33:59.792353 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:59.791183 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1abdaf-pd6n2"] Sep 10 16:33:59.835436 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:33:59.835395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:00.145077 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:00.145024 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:00.222427 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:00.222394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:00.938092 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:00.938055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:01.600989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:01.600954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:01.758101 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:01.758065 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b68f7754-c17b-44ab-89b7-1ece7f8d16ea" path="/var/lib/kubelet/pods/b68f7754-c17b-44ab-89b7-1ece7f8d16ea/volumes" Sep 10 16:34:01.862218 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:34:01.862114 2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-496136/noten-model-496136-kserve-self-signed-certs: secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:34:01.862218 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:34:01.862205 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs podName:e3657142-31a5-44ae-8705-092bf19f5ec9 nodeName:}" failed. No retries permitted until 2026-09-10 16:34:17.862185189 +0000 UTC m=+1008.588504100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs") pod "noten-model-496136-kserve-5446b447d7-jhk84" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9") : secret "noten-model-496136-kserve-self-signed-certs" not found Sep 10 16:34:02.047774 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.047660 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" podUID="e3657142-31a5-44ae-8705-092bf19f5ec9" containerName="main" containerID="cri-o://17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a" gracePeriod=45 Sep 10 16:34:02.326943 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.326918 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.369983 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-home\") pod \"e3657142-31a5-44ae-8705-092bf19f5ec9\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.370076 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-tmp-dir\") pod \"e3657142-31a5-44ae-8705-092bf19f5ec9\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.370116 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs\") pod \"e3657142-31a5-44ae-8705-092bf19f5ec9\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.370198 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-dshm\") pod \"e3657142-31a5-44ae-8705-092bf19f5ec9\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.370230 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km4m8\" (UniqueName: \"kubernetes.io/projected/e3657142-31a5-44ae-8705-092bf19f5ec9-kube-api-access-km4m8\") pod \"e3657142-31a5-44ae-8705-092bf19f5ec9\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.370299 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-model-cache\") pod \"e3657142-31a5-44ae-8705-092bf19f5ec9\" (UID: \"e3657142-31a5-44ae-8705-092bf19f5ec9\") " Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.370432 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-home" (OuterVolumeSpecName: "home") pod "e3657142-31a5-44ae-8705-092bf19f5ec9" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.370639 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3657142-31a5-44ae-8705-092bf19f5ec9" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:02.371063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.370684 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-home\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:02.371823 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.371790 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3657142-31a5-44ae-8705-092bf19f5ec9" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:02.373362 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.373337 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-dshm" (OuterVolumeSpecName: "dshm") pod "e3657142-31a5-44ae-8705-092bf19f5ec9" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:02.373586 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.373560 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3657142-31a5-44ae-8705-092bf19f5ec9-kube-api-access-km4m8" (OuterVolumeSpecName: "kube-api-access-km4m8") pod "e3657142-31a5-44ae-8705-092bf19f5ec9" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9"). InnerVolumeSpecName "kube-api-access-km4m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:02.373827 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.373809 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3657142-31a5-44ae-8705-092bf19f5ec9" (UID: "e3657142-31a5-44ae-8705-092bf19f5ec9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:34:02.471831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.471730 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-dshm\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:02.471831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.471767 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4m8\" (UniqueName: \"kubernetes.io/projected/e3657142-31a5-44ae-8705-092bf19f5ec9-kube-api-access-km4m8\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:02.471831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.471779 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-model-cache\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:02.471831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.471789 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3657142-31a5-44ae-8705-092bf19f5ec9-tmp-dir\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:02.471831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:02.471796 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3657142-31a5-44ae-8705-092bf19f5ec9-tls-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:03.135842 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.135803 2562 generic.go:356] "Generic (PLEG): container finished" podID="e3657142-31a5-44ae-8705-092bf19f5ec9" containerID="17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a" exitCode=0 Sep 10 16:34:03.136271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.135854 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" Sep 10 16:34:03.136271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.135892 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" event={"ID":"e3657142-31a5-44ae-8705-092bf19f5ec9","Type":"ContainerDied","Data":"17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a"} Sep 10 16:34:03.136271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.135942 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84" event={"ID":"e3657142-31a5-44ae-8705-092bf19f5ec9","Type":"ContainerDied","Data":"f7a7b015fec068d1ff0a505d368a63f1225af8353bfd6cd1fed0c5592a160471"} Sep 10 16:34:03.136271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.135964 2562 scope.go:117] "RemoveContainer" containerID="17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a" Sep 10 16:34:03.167096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.167067 2562 scope.go:117] "RemoveContainer" containerID="17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a" Sep 10 16:34:03.167673 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:34:03.167632 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a\": container with ID starting with 17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a not found: ID does not exist" containerID="17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a" Sep 10 16:34:03.167823 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.167671 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a"} err="failed to get container status \"17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a\": rpc error: code = NotFound desc = could not find container \"17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a\": container with ID starting with 17a8ec09f0906c502ac7a7d35cd30d13d7423521665647f89292f33c5c86282a not found: ID does not exist" Sep 10 16:34:03.184164 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.184132 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84"] Sep 10 16:34:03.188331 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.188301 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-496136/noten-model-496136-kserve-5446b447d7-jhk84"] Sep 10 16:34:03.762815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:03.762766 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3657142-31a5-44ae-8705-092bf19f5ec9" path="/var/lib/kubelet/pods/e3657142-31a5-44ae-8705-092bf19f5ec9/volumes" Sep 10 16:34:06.619418 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:06.619389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:06.687105 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:06.687067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:06.882341 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:06.882249 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:07.451770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:07.451741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:07.846171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:07.846138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:07.946369 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:07.946334 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:08.154404 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:08.154317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:08.939618 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:08.939581 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:09.033222 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:09.033183 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:34:09.108180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:09.108147 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:34:09.108349 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:09.108215 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:34:09.643596 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:09.643561 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:09.744861 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:09.744831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:10.115210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:10.115172 2562 patch_prober.go:28] interesting pod/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 16:34:10.115669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:10.115241 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 16:34:10.387535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:10.387449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:10.478869 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:10.478834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:10.579314 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:10.579276 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:10.804527 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:10.804448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:11.207891 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.207857 2562 generic.go:356] "Generic (PLEG): container finished" podID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerID="74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128" exitCode=0 Sep 10 16:34:11.208306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.207934 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" event={"ID":"a5ae9684-7dd5-42fd-a345-b4cb23d150a6","Type":"ContainerDied","Data":"74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128"} Sep 10 16:34:11.208425 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.208409 2562 scope.go:117] "RemoveContainer" containerID="74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128" Sep 10 16:34:11.209655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.209625 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:11.209727 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.209703 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:11.209762 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.209738 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:11.297664 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.297632 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:11.446665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.446619 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:11.646332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:11.646289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:12.215666 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:12.215635 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" event={"ID":"a5ae9684-7dd5-42fd-a345-b4cb23d150a6","Type":"ContainerStarted","Data":"33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d"} Sep 10 16:34:12.837022 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:12.836984 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:34:12.837264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:12.837242 2562 patch_prober.go:28] interesting pod/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 16:34:12.837320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:12.837290 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 16:34:13.837799 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:13.837765 2562 patch_prober.go:28] interesting pod/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 16:34:13.838269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:13.837824 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 16:34:14.837232 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:14.837191 2562 patch_prober.go:28] interesting pod/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 16:34:14.837420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:14.837258 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 16:34:15.837569 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:15.837533 2562 patch_prober.go:28] interesting pod/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 16:34:15.837957 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:15.837589 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 16:34:16.837271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:16.837229 2562 patch_prober.go:28] interesting pod/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 16:34:16.837460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:16.837290 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 16:34:16.950300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:16.950210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:17.012448 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:17.012412 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:17.249571 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:17.249478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:17.841327 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:17.841294 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:34:17.841697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:17.841677 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:34:17.842432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:17.842413 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:34:17.965261 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:17.965228 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:17.972712 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:17.972679 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g"] Sep 10 16:34:17.975994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:17.975958 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6082-29817630-68c9g"] Sep 10 16:34:18.030982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:18.030945 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:18.092716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:18.092626 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:18.164028 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:18.163986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:18.245081 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:18.245027 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:18.501534 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:18.501503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:18.894411 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:18.894287 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:19.582955 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:19.582918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:19.760112 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:19.760070 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85217ea0-360b-48c5-b35d-45413c837d9c" path="/var/lib/kubelet/pods/85217ea0-360b-48c5-b35d-45413c837d9c/volumes" Sep 10 16:34:19.928024 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:19.927926 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:21.336144 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:21.336108 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:34:24.734130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:24.734095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:24.792482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:24.792456 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:24.997983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:24.997186 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh"] Sep 10 16:34:24.997983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:24.997440 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" containerID="cri-o://51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a" gracePeriod=30 Sep 10 16:34:25.009535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:25.009500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:25.586031 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:25.585992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:25.878882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:25.878797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:26.096462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:26.096431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:26.497417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:26.497385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:27.041716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.041679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:27.063853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.063822 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx"] Sep 10 16:34:27.064138 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.064113 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" podUID="af090985-439e-4c1b-aee6-04af76d70104" containerName="payload-processing" containerID="cri-o://3bd9f3fc47ccfe79dff7e8137b5847874682b3cc0d34b4dfc615c61950de9764" gracePeriod=30 Sep 10 16:34:27.074264 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.074238 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9"] Sep 10 16:34:27.074653 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.074619 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" podUID="fba14a01-412a-4116-b65e-98cee7191cbc" containerName="payload-pre-processing" containerID="cri-o://e333dd9ff70339e17fcc1cc11813e9d3561e36e54d939ea36666b61cb912f095" gracePeriod=30 Sep 10 16:34:27.109414 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.109372 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:27.167895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.167855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:27.230234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.230195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:27.319180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.318796 2562 generic.go:356] "Generic (PLEG): container finished" podID="af090985-439e-4c1b-aee6-04af76d70104" containerID="3bd9f3fc47ccfe79dff7e8137b5847874682b3cc0d34b4dfc615c61950de9764" exitCode=0 Sep 10 16:34:27.319180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.318956 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" event={"ID":"af090985-439e-4c1b-aee6-04af76d70104","Type":"ContainerDied","Data":"3bd9f3fc47ccfe79dff7e8137b5847874682b3cc0d34b4dfc615c61950de9764"} Sep 10 16:34:27.322110 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.321844 2562 generic.go:356] "Generic (PLEG): container finished" podID="fba14a01-412a-4116-b65e-98cee7191cbc" containerID="e333dd9ff70339e17fcc1cc11813e9d3561e36e54d939ea36666b61cb912f095" exitCode=0 Sep 10 16:34:27.322110 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.321898 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" event={"ID":"fba14a01-412a-4116-b65e-98cee7191cbc","Type":"ContainerDied","Data":"e333dd9ff70339e17fcc1cc11813e9d3561e36e54d939ea36666b61cb912f095"} Sep 10 16:34:27.358560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.358535 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:34:27.372413 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.372384 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:34:27.439463 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.439422 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af090985-439e-4c1b-aee6-04af76d70104-plugins-config-volume\") pod \"af090985-439e-4c1b-aee6-04af76d70104\" (UID: \"af090985-439e-4c1b-aee6-04af76d70104\") " Sep 10 16:34:27.439755 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.439501 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l54jh\" (UniqueName: \"kubernetes.io/projected/af090985-439e-4c1b-aee6-04af76d70104-kube-api-access-l54jh\") pod \"af090985-439e-4c1b-aee6-04af76d70104\" (UID: \"af090985-439e-4c1b-aee6-04af76d70104\") " Sep 10 16:34:27.439755 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.439528 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dwxw\" (UniqueName: \"kubernetes.io/projected/fba14a01-412a-4116-b65e-98cee7191cbc-kube-api-access-5dwxw\") pod \"fba14a01-412a-4116-b65e-98cee7191cbc\" (UID: \"fba14a01-412a-4116-b65e-98cee7191cbc\") " Sep 10 16:34:27.439755 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.439555 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba14a01-412a-4116-b65e-98cee7191cbc-plugins-config-volume\") pod \"fba14a01-412a-4116-b65e-98cee7191cbc\" (UID: \"fba14a01-412a-4116-b65e-98cee7191cbc\") " Sep 10 16:34:27.439952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.439930 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af090985-439e-4c1b-aee6-04af76d70104-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af090985-439e-4c1b-aee6-04af76d70104" (UID: "af090985-439e-4c1b-aee6-04af76d70104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:34:27.440096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.440075 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af090985-439e-4c1b-aee6-04af76d70104-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:27.440337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.440309 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba14a01-412a-4116-b65e-98cee7191cbc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fba14a01-412a-4116-b65e-98cee7191cbc" (UID: "fba14a01-412a-4116-b65e-98cee7191cbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:34:27.441867 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.441836 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba14a01-412a-4116-b65e-98cee7191cbc-kube-api-access-5dwxw" (OuterVolumeSpecName: "kube-api-access-5dwxw") pod "fba14a01-412a-4116-b65e-98cee7191cbc" (UID: "fba14a01-412a-4116-b65e-98cee7191cbc"). InnerVolumeSpecName "kube-api-access-5dwxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:27.442587 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.442546 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af090985-439e-4c1b-aee6-04af76d70104-kube-api-access-l54jh" (OuterVolumeSpecName: "kube-api-access-l54jh") pod "af090985-439e-4c1b-aee6-04af76d70104" (UID: "af090985-439e-4c1b-aee6-04af76d70104"). InnerVolumeSpecName "kube-api-access-l54jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:27.541633 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.541588 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l54jh\" (UniqueName: \"kubernetes.io/projected/af090985-439e-4c1b-aee6-04af76d70104-kube-api-access-l54jh\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:27.541633 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.541624 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dwxw\" (UniqueName: \"kubernetes.io/projected/fba14a01-412a-4116-b65e-98cee7191cbc-kube-api-access-5dwxw\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:27.541633 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.541633 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba14a01-412a-4116-b65e-98cee7191cbc-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:27.600544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.600453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:27.709988 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:27.709855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:28.329049 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.329006 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" Sep 10 16:34:28.329622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.329063 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9" event={"ID":"fba14a01-412a-4116-b65e-98cee7191cbc","Type":"ContainerDied","Data":"229d4d0633e6cfc614f0efcb43d900b1fea10b1c65ffcac54afd54c0fe26d47e"} Sep 10 16:34:28.329622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.329134 2562 scope.go:117] "RemoveContainer" containerID="e333dd9ff70339e17fcc1cc11813e9d3561e36e54d939ea36666b61cb912f095" Sep 10 16:34:28.330800 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.330766 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" event={"ID":"af090985-439e-4c1b-aee6-04af76d70104","Type":"ContainerDied","Data":"cc31a9621274e5a8011fb5da06f1338b43d1a8cefa2fcbb9989b6c7de170738a"} Sep 10 16:34:28.330926 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.330777 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx" Sep 10 16:34:28.338358 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.338340 2562 scope.go:117] "RemoveContainer" containerID="3bd9f3fc47ccfe79dff7e8137b5847874682b3cc0d34b4dfc615c61950de9764" Sep 10 16:34:28.351601 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.351576 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9"] Sep 10 16:34:28.356823 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.356797 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e77069-b556c8b58-qsgx9"] Sep 10 16:34:28.367690 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.367661 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx"] Sep 10 16:34:28.371285 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:28.371262 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6e77069-66d4c8cfbf-8n5vx"] Sep 10 16:34:29.143234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:29.143197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:29.159648 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:29.159609 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:34:29.238380 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:29.238350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:29.757754 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:29.757722 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af090985-439e-4c1b-aee6-04af76d70104" path="/var/lib/kubelet/pods/af090985-439e-4c1b-aee6-04af76d70104/volumes" Sep 10 16:34:29.758180 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:29.758165 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fba14a01-412a-4116-b65e-98cee7191cbc" path="/var/lib/kubelet/pods/fba14a01-412a-4116-b65e-98cee7191cbc/volumes" Sep 10 16:34:30.252381 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.252353 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:34:30.268776 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268640 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-data\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.268776 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268695 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-podinfo\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.268776 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268731 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-token\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.268776 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268781 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-socket\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.269142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268812 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dc34e2-1be0-49c6-88dc-fd83453b041f-istiod-ca-cert\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.269142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268852 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-certs\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.269142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268927 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-envoy\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.269142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268936 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-data" (OuterVolumeSpecName: "istio-data") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:30.269142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.268980 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-credential-socket\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.269142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.269007 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr4ww\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-kube-api-access-lr4ww\") pod \"14dc34e2-1be0-49c6-88dc-fd83453b041f\" (UID: \"14dc34e2-1be0-49c6-88dc-fd83453b041f\") " Sep 10 16:34:30.269450 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.269186 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:30.269450 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.269321 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14dc34e2-1be0-49c6-88dc-fd83453b041f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:34:30.269554 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.269538 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:30.269610 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.269583 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.269610 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.269604 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dc34e2-1be0-49c6-88dc-fd83453b041f-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.269728 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.269618 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.269848 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.269818 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:30.271899 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.271858 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:30.272100 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.272074 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-token" (OuterVolumeSpecName: "istio-token") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:30.272190 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.272095 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-kube-api-access-lr4ww" (OuterVolumeSpecName: "kube-api-access-lr4ww") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "kube-api-access-lr4ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:30.272190 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.272151 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14dc34e2-1be0-49c6-88dc-fd83453b041f" (UID: "14dc34e2-1be0-49c6-88dc-fd83453b041f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:34:30.300646 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.300614 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:30.346347 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.346313 2562 generic.go:356] "Generic (PLEG): container finished" podID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerID="51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a" exitCode=0 Sep 10 16:34:30.346515 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.346362 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" event={"ID":"14dc34e2-1be0-49c6-88dc-fd83453b041f","Type":"ContainerDied","Data":"51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a"} Sep 10 16:34:30.346515 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.346368 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" Sep 10 16:34:30.346515 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.346432 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh" event={"ID":"14dc34e2-1be0-49c6-88dc-fd83453b041f","Type":"ContainerDied","Data":"1aa2e7a294f2a8f05704644a822a216b02f897e263bc1c48e23682f18258aa04"} Sep 10 16:34:30.346515 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.346456 2562 scope.go:117] "RemoveContainer" containerID="51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a" Sep 10 16:34:30.357105 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.357079 2562 scope.go:117] "RemoveContainer" containerID="905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a" Sep 10 16:34:30.361716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.361691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:30.366662 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.366638 2562 scope.go:117] "RemoveContainer" containerID="51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a" Sep 10 16:34:30.367613 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:34:30.367584 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a\": container with ID starting with 51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a not found: ID does not exist" containerID="51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a" Sep 10 16:34:30.368008 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.367626 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a"} err="failed to get container status \"51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a\": rpc error: code = NotFound desc = could not find container \"51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a\": container with ID starting with 51821645d72ade8819995e519f0b68f17d380d93e9eac2426a7d5deb0b76995a not found: ID does not exist" Sep 10 16:34:30.368008 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.367653 2562 scope.go:117] "RemoveContainer" containerID="905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a" Sep 10 16:34:30.368008 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:34:30.367993 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a\": container with ID starting with 905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a not found: ID does not exist" containerID="905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a" Sep 10 16:34:30.368267 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.368022 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a"} err="failed to get container status \"905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a\": rpc error: code = NotFound desc = could not find container \"905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a\": container with ID starting with 905db3b49d2ac087a475c5fef1c55a6ea6ff5081bd8ba63e5a800a4d63c9ee4a not found: ID does not exist" Sep 10 16:34:30.370373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.370346 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.370373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.370373 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr4ww\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-kube-api-access-lr4ww\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.370548 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.370384 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.370548 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.370393 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.370548 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.370401 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.370548 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.370408 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dc34e2-1be0-49c6-88dc-fd83453b041f-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:30.370754 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.370569 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh"] Sep 10 16:34:30.372541 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.372516 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6082-openshift-default-6c54d795cd-mdzqh"] Sep 10 16:34:30.463335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.463298 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:30.764707 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.764678 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:30.862851 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.862815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:30.913609 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.913560 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6"] Sep 10 16:34:30.914099 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914079 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba14a01-412a-4116-b65e-98cee7191cbc" containerName="payload-pre-processing" Sep 10 16:34:30.914209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914100 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba14a01-412a-4116-b65e-98cee7191cbc" containerName="payload-pre-processing" Sep 10 16:34:30.914209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914137 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" Sep 10 16:34:30.914209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914146 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" Sep 10 16:34:30.914209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914165 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3657142-31a5-44ae-8705-092bf19f5ec9" containerName="main" Sep 10 16:34:30.914209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914172 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3657142-31a5-44ae-8705-092bf19f5ec9" containerName="main" Sep 10 16:34:30.914209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914190 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af090985-439e-4c1b-aee6-04af76d70104" containerName="payload-processing" Sep 10 16:34:30.914209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914199 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af090985-439e-4c1b-aee6-04af76d70104" containerName="payload-processing" Sep 10 16:34:30.914560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914292 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" Sep 10 16:34:30.914560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914309 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" Sep 10 16:34:30.914560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914319 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba14a01-412a-4116-b65e-98cee7191cbc" containerName="payload-pre-processing" Sep 10 16:34:30.914560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914329 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="af090985-439e-4c1b-aee6-04af76d70104" containerName="payload-processing" Sep 10 16:34:30.914560 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.914341 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3657142-31a5-44ae-8705-092bf19f5ec9" containerName="main" Sep 10 16:34:30.918973 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.918951 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:30.921936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.921912 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1925d8\"" Sep 10 16:34:30.922057 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.921953 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1925d8-dockercfg-9n7gq\"" Sep 10 16:34:30.927808 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.927777 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6"] Sep 10 16:34:30.928885 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.928860 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:30.977618 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.977576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46z7\" (UniqueName: \"kubernetes.io/projected/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-kube-api-access-j46z7\") pod \"payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6\" (UID: \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:30.977791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:30.977683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6\" (UID: \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:31.078407 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:31.078314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j46z7\" (UniqueName: \"kubernetes.io/projected/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-kube-api-access-j46z7\") pod \"payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6\" (UID: \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:31.078407 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:31.078385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6\" (UID: \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:31.079414 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:31.079388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6\" (UID: \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:31.087904 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:31.087879 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46z7\" (UniqueName: \"kubernetes.io/projected/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-kube-api-access-j46z7\") pod \"payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6\" (UID: \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:31.231804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:31.231765 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:31.503578 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:31.503554 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6"] Sep 10 16:34:31.505790 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:34:31.505760 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcda90f_ddd8_4aae_808d_12fe98ac6cc1.slice/crio-cc9ef44cf0d9ff8d0e1cc76b66acdebe0654ea1de34ac34b03d68f92459e72c8 WatchSource:0}: Error finding container cc9ef44cf0d9ff8d0e1cc76b66acdebe0654ea1de34ac34b03d68f92459e72c8: Status 404 returned error can't find the container with id cc9ef44cf0d9ff8d0e1cc76b66acdebe0654ea1de34ac34b03d68f92459e72c8 Sep 10 16:34:31.759687 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:31.759602 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" path="/var/lib/kubelet/pods/14dc34e2-1be0-49c6-88dc-fd83453b041f/volumes" Sep 10 16:34:32.362122 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:32.362084 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" event={"ID":"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1","Type":"ContainerStarted","Data":"9a899e10d6de62d0017dd765f0b7b70c6ccada04312bf9fca87d50352cee4204"} Sep 10 16:34:32.362506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:32.362139 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" event={"ID":"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1","Type":"ContainerStarted","Data":"cc9ef44cf0d9ff8d0e1cc76b66acdebe0654ea1de34ac34b03d68f92459e72c8"} Sep 10 16:34:32.362506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:32.362166 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:32.383577 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:32.383520 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" podStartSLOduration=2.383505068 podStartE2EDuration="2.383505068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:34:32.381539246 +0000 UTC m=+1023.107858177" watchObservedRunningTime="2026-09-10 16:34:32.383505068 +0000 UTC m=+1023.109823998" Sep 10 16:34:33.665615 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:33.665581 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:33.766088 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:33.766029 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:33.909550 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:33.909518 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr"] Sep 10 16:34:33.909755 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:33.909733 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" containerID="cri-o://33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d" gracePeriod=30 Sep 10 16:34:33.944908 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:33.944822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:34.281863 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:34.281823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:34.380579 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:34.380538 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:34.485091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:34.485060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:38.857156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:38.857119 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:34:39.167932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.167905 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:34:39.263233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263175 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-credential-socket\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263231 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-envoy\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263287 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v94m\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-kube-api-access-7v94m\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263308 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-podinfo\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263343 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-certs\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263373 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istiod-ca-cert\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263410 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-socket\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263452 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-data\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263504 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-token\") pod \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\" (UID: \"a5ae9684-7dd5-42fd-a345-b4cb23d150a6\") " Sep 10 16:34:39.263791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263502 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:39.263791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263764 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:39.263926 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263780 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:39.263926 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263795 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.263926 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.263867 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:34:39.264078 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.264014 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:39.265744 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.265712 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-kube-api-access-7v94m" (OuterVolumeSpecName: "kube-api-access-7v94m") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "kube-api-access-7v94m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:39.265866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.265745 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:34:39.265866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.265805 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:34:39.265866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.265845 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5ae9684-7dd5-42fd-a345-b4cb23d150a6" (UID: "a5ae9684-7dd5-42fd-a345-b4cb23d150a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:34:39.364798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.364756 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.364798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.364789 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.364798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.364800 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v94m\" (UniqueName: \"kubernetes.io/projected/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-kube-api-access-7v94m\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.364798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.364810 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.365130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.364818 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.365130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.364826 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.365130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.364834 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.365130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.364841 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ae9684-7dd5-42fd-a345-b4cb23d150a6-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:34:39.406406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.406317 2562 generic.go:356] "Generic (PLEG): container finished" podID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerID="33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d" exitCode=0 Sep 10 16:34:39.406406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.406365 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" Sep 10 16:34:39.406582 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.406405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" event={"ID":"a5ae9684-7dd5-42fd-a345-b4cb23d150a6","Type":"ContainerDied","Data":"33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d"} Sep 10 16:34:39.406582 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.406464 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr" event={"ID":"a5ae9684-7dd5-42fd-a345-b4cb23d150a6","Type":"ContainerDied","Data":"d09378be3c11477f7b1f92ecfc64db889b98124dad6ef7313f7b2261e6b857d8"} Sep 10 16:34:39.406582 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.406487 2562 scope.go:117] "RemoveContainer" containerID="33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d" Sep 10 16:34:39.416645 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.416626 2562 scope.go:117] "RemoveContainer" containerID="74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128" Sep 10 16:34:39.424931 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.424900 2562 scope.go:117] "RemoveContainer" containerID="33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d" Sep 10 16:34:39.425304 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:34:39.425275 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d\": container with ID starting with 33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d not found: ID does not exist" containerID="33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d" Sep 10 16:34:39.426145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.425622 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d"} err="failed to get container status \"33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d\": rpc error: code = NotFound desc = could not find container \"33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d\": container with ID starting with 33943628285caab341fdcb0eba2c83ea76b5ee85c1802bf0a7090a6f8191ac5d not found: ID does not exist" Sep 10 16:34:39.426145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.426142 2562 scope.go:117] "RemoveContainer" containerID="74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128" Sep 10 16:34:39.426577 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:34:39.426549 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128\": container with ID starting with 74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128 not found: ID does not exist" containerID="74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128" Sep 10 16:34:39.426637 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.426587 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128"} err="failed to get container status \"74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128\": rpc error: code = NotFound desc = could not find container \"74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128\": container with ID starting with 74fe73cd926d6e9975252690079cc534035defb713f0d882fcddc0ae5e8e9128 not found: ID does not exist" Sep 10 16:34:39.428197 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.428176 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr"] Sep 10 16:34:39.431498 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.431476 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e6e77069-openshift-default-6f748ccc84-4rjxr"] Sep 10 16:34:39.757616 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:39.757580 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" path="/var/lib/kubelet/pods/a5ae9684-7dd5-42fd-a345-b4cb23d150a6/volumes" Sep 10 16:34:40.755852 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:40.755820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:40.832894 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:40.832837 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:40.943372 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:40.943341 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:41.044001 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:41.043919 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:41.146300 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:41.146267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:43.373980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:43.373950 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:34:46.987793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:46.987739 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:47.090642 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:47.090611 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:47.863121 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:47.863074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:48.136049 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.135943 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg"] Sep 10 16:34:48.136648 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.136569 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" Sep 10 16:34:48.136648 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.136590 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" Sep 10 16:34:48.136648 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.136621 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" Sep 10 16:34:48.136648 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.136630 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" Sep 10 16:34:48.136907 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.136654 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" Sep 10 16:34:48.136907 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.136664 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="14dc34e2-1be0-49c6-88dc-fd83453b041f" containerName="istio-proxy" Sep 10 16:34:48.136907 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.136801 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" Sep 10 16:34:48.136907 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.136817 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ae9684-7dd5-42fd-a345-b4cb23d150a6" containerName="istio-proxy" Sep 10 16:34:48.143793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.143760 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.147497 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.147466 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-bd1ece-openshift-default-dockercfg-8c5qj\"" Sep 10 16:34:48.151911 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.151882 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg"] Sep 10 16:34:48.166431 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.166403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:48.248939 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.248894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-envoy\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.249161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.248955 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-token\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.249161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.249076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-credential-socket\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.249161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.249116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-certs\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.249161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.249138 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istiod-ca-cert\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.249365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.249265 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-data\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.249365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.249323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rcc\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-kube-api-access-75rcc\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.249365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.249353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-socket\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.249505 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.249401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-podinfo\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.349973 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.349923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-data\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.349993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rcc\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-kube-api-access-75rcc\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-socket\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-podinfo\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350185 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-envoy\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-token\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350346 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-data\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-socket\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350575 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350457 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-credential-socket\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350575 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-certs\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350575 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istiod-ca-cert\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350736 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350701 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-credential-socket\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.350837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.350815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-certs\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.351298 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.351276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istiod-ca-cert\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.352397 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.352369 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-envoy\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.352680 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.352661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-podinfo\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.358464 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.358434 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-token\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.359032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.359015 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rcc\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-kube-api-access-75rcc\") pod \"e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.458455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.458364 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:48.609918 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.609880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:48.737193 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.737164 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg"] Sep 10 16:34:48.739293 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:34:48.739263 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c08a94d_1dd0_45c1_8821_2d6b930c0e95.slice/crio-68a2fe37fbac851dd26c4b5399855afee060e40ccd16d45df9690c9cb13c03b0 WatchSource:0}: Error finding container 68a2fe37fbac851dd26c4b5399855afee060e40ccd16d45df9690c9cb13c03b0: Status 404 returned error can't find the container with id 68a2fe37fbac851dd26c4b5399855afee060e40ccd16d45df9690c9cb13c03b0 Sep 10 16:34:48.741598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.741561 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:48.741701 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.741645 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:48.741701 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:48.741691 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:49.463517 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:49.463479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" event={"ID":"1c08a94d-1dd0-45c1-8821-2d6b930c0e95","Type":"ContainerStarted","Data":"aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc"} Sep 10 16:34:49.464073 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:49.463532 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" event={"ID":"1c08a94d-1dd0-45c1-8821-2d6b930c0e95","Type":"ContainerStarted","Data":"68a2fe37fbac851dd26c4b5399855afee060e40ccd16d45df9690c9cb13c03b0"} Sep 10 16:34:49.486326 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:49.486273 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" podStartSLOduration=1.486255626 podStartE2EDuration="1.486255626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:34:49.484929846 +0000 UTC m=+1040.211248777" watchObservedRunningTime="2026-09-10 16:34:49.486255626 +0000 UTC m=+1040.212574556" Sep 10 16:34:50.459368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:50.459328 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:50.464702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:50.464676 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:50.468364 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:50.468340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:50.469223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:50.469206 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:34:50.714706 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:50.714606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:51.562269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:51.562223 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:51.651931 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:51.651896 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:52.578179 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:52.578142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:53.289757 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:53.289721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.006837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.006803 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.076161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.076126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.145072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.144700 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.335335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.335234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.439733 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.439702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.558609 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.558575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.635005 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.634914 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc"] Sep 10 16:34:54.639419 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.639393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.642628 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.642603 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3ccbd4-openshift-default-dockercfg-ssl6c\"" Sep 10 16:34:54.653197 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.653170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.654155 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.654134 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc"] Sep 10 16:34:54.742638 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.742600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.817401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817367 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-envoy\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.817401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-credential-socket\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.817678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817496 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-data\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.817678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-token\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.817678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-podinfo\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.817678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istiod-ca-cert\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.817892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-socket\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.817892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98tt\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-kube-api-access-b98tt\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.817892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.817756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-certs\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.832344 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.832309 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:54.919328 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istiod-ca-cert\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919328 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-socket\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919328 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98tt\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-kube-api-access-b98tt\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-certs\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919392 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-envoy\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-credential-socket\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-data\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-token\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919576 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-podinfo\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919904 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919733 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-socket\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.919904 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-certs\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.920011 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.919936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-data\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.920120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.920109 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istiod-ca-cert\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.920213 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.920193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-credential-socket\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.921859 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.921834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-envoy\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.922021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.921994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-podinfo\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.929373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.929352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-token\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.929574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.929556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98tt\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-kube-api-access-b98tt\") pod \"e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:54.953425 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:54.953396 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:55.085707 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.085672 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:55.233792 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.233761 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc"] Sep 10 16:34:55.236683 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:34:55.236652 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6b2e4f_73bd_4958_aafa_f8f5967d9659.slice/crio-d1693713f25439165d8e62ebed45fb9b2cb35137fb704ac8e4a9ae11098a7519 WatchSource:0}: Error finding container d1693713f25439165d8e62ebed45fb9b2cb35137fb704ac8e4a9ae11098a7519: Status 404 returned error can't find the container with id d1693713f25439165d8e62ebed45fb9b2cb35137fb704ac8e4a9ae11098a7519 Sep 10 16:34:55.239218 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.239179 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:55.239302 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.239265 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:55.239341 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.239315 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:34:55.498674 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.498641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" event={"ID":"9d6b2e4f-73bd-4958-aafa-f8f5967d9659","Type":"ContainerStarted","Data":"7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725"} Sep 10 16:34:55.498857 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.498702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" event={"ID":"9d6b2e4f-73bd-4958-aafa-f8f5967d9659","Type":"ContainerStarted","Data":"d1693713f25439165d8e62ebed45fb9b2cb35137fb704ac8e4a9ae11098a7519"} Sep 10 16:34:55.519406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.519355 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podStartSLOduration=1.51933524 podStartE2EDuration="1.51933524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:34:55.51821852 +0000 UTC m=+1046.244537478" watchObservedRunningTime="2026-09-10 16:34:55.51933524 +0000 UTC m=+1046.245654160" Sep 10 16:34:55.764686 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.764603 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:55.954518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:55.954484 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:56.786427 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:56.786383 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:56.959481 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:56.959451 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:57.415622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:57.415585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:57.509553 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:57.509519 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:57.517377 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:57.511421 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:34:57.519123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:57.519097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:57.616350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:57.616317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:58.241898 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:58.241859 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:34:58.266612 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:58.266575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:58.971461 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:58.971431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:59.072244 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:59.072190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:59.171476 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:59.171442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:59.561086 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:59.559880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:59.648290 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:59.648195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:34:59.970616 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:34:59.970501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:00.705028 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:00.704967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:01.450458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:01.450424 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:06.510308 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:06.510251 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 16:35:06.510701 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:06.510343 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 16:35:08.580310 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:08.580280 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerID="aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc" exitCode=0 Sep 10 16:35:08.580751 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:08.580318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" event={"ID":"1c08a94d-1dd0-45c1-8821-2d6b930c0e95","Type":"ContainerDied","Data":"aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc"} Sep 10 16:35:08.580751 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:08.580678 2562 scope.go:117] "RemoveContainer" containerID="aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc" Sep 10 16:35:08.581738 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:08.581709 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:08.581800 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:08.581778 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:08.581842 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:08.581808 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:09.593581 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:09.593529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" event={"ID":"1c08a94d-1dd0-45c1-8821-2d6b930c0e95","Type":"ContainerStarted","Data":"aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff"} Sep 10 16:35:10.459398 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:10.459356 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:35:10.459626 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:10.459517 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:35:10.459626 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:10.459580 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 16:35:10.640247 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:10.640210 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:35:11.459607 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:11.459561 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:35:11.459938 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:11.459901 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 16:35:12.460009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:12.459442 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:35:12.460009 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:12.459533 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 16:35:13.459518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:13.459067 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:35:13.459518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:13.459130 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 16:35:13.552291 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:13.552244 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 16:35:13.552867 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:13.552832 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 16:35:14.459361 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:14.459265 2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 16:35:14.459643 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:14.459605 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 16:35:14.947633 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:14.947593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:15.462604 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:15.462564 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:35:15.462902 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:15.462874 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:35:15.463542 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:15.463525 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:35:16.253243 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.253203 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:16.371551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.371497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:16.640970 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.640882 2562 generic.go:356] "Generic (PLEG): container finished" podID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerID="7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725" exitCode=0 Sep 10 16:35:16.640970 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.640949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" event={"ID":"9d6b2e4f-73bd-4958-aafa-f8f5967d9659","Type":"ContainerDied","Data":"7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725"} Sep 10 16:35:16.641395 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.641380 2562 scope.go:117] "RemoveContainer" containerID="7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725" Sep 10 16:35:16.642412 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.642386 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:16.642479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.642454 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:16.642523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.642503 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:16.691796 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.691726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:16.945823 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.945691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:16.975866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.975832 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx"] Sep 10 16:35:16.978862 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.978838 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:16.982067 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.982017 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-035bd5-openshift-default-dockercfg-wgnqn\"" Sep 10 16:35:16.993329 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:16.993149 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx"] Sep 10 16:35:17.046246 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.046215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:17.069385 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069211 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-envoy\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.069385 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t9h5\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-kube-api-access-5t9h5\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.069385 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069343 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-token\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.069385 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-socket\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.069784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istiod-ca-cert\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.069784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-podinfo\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.069784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-certs\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.069784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-credential-socket\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.069784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.069632 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-data\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170344 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-certs\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170344 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170342 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-credential-socket\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-data\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-envoy\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t9h5\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-kube-api-access-5t9h5\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-token\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-socket\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istiod-ca-cert\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170894 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-podinfo\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170894 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-certs\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.170894 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170823 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-credential-socket\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.171023 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-data\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.171023 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.170990 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-socket\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.171405 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.171385 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istiod-ca-cert\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.173082 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.173027 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-envoy\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.173201 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.173183 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-podinfo\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.179761 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.179731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-token\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.180105 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.180086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t9h5\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-kube-api-access-5t9h5\") pod \"e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.291381 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.291348 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:17.549950 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.549841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:17.574954 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.574921 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx"] Sep 10 16:35:17.576378 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:35:17.576337 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57edd1e_1d1a_479a_8d8a_657e9cfc434d.slice/crio-bf075147563291e149c2cf09a4d54e3512c7ea4fca7623ac12c89e18ee8e83a6 WatchSource:0}: Error finding container bf075147563291e149c2cf09a4d54e3512c7ea4fca7623ac12c89e18ee8e83a6: Status 404 returned error can't find the container with id bf075147563291e149c2cf09a4d54e3512c7ea4fca7623ac12c89e18ee8e83a6 Sep 10 16:35:17.578965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.578926 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:17.579078 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.579001 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:17.579078 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.579055 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:17.645765 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.645730 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:17.652268 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.652224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" event={"ID":"9d6b2e4f-73bd-4958-aafa-f8f5967d9659","Type":"ContainerStarted","Data":"24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86"} Sep 10 16:35:17.655545 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.654735 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" event={"ID":"c57edd1e-1d1a-479a-8d8a-657e9cfc434d","Type":"ContainerStarted","Data":"bf075147563291e149c2cf09a4d54e3512c7ea4fca7623ac12c89e18ee8e83a6"} Sep 10 16:35:17.697965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.697891 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" podStartSLOduration=1.697868599 podStartE2EDuration="1.697868599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:35:17.696410962 +0000 UTC m=+1068.422729910" watchObservedRunningTime="2026-09-10 16:35:17.697868599 +0000 UTC m=+1068.424187529" Sep 10 16:35:17.954960 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.954857 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:35:17.955432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.955288 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:35:17.955620 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:17.955595 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 16:35:18.291840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:18.291792 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:18.297337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:18.297309 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:18.389301 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:18.389264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:18.666568 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:18.666445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" event={"ID":"c57edd1e-1d1a-479a-8d8a-657e9cfc434d","Type":"ContainerStarted","Data":"9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8"} Sep 10 16:35:18.666735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:18.666621 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:18.667766 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:18.667742 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:18.954669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:18.954555 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:35:18.954669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:18.954650 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 16:35:19.066127 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:19.066081 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:19.587494 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:19.587437 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:19.688061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:19.688010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:19.898323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:19.898230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:19.954805 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:19.954762 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:35:19.955003 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:19.954835 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 16:35:20.521410 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:20.521375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:20.954811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:20.954294 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:35:20.954811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:20.954371 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 16:35:21.281350 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.281300 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:21.380082 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.380017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:21.813562 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.813464 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp"] Sep 10 16:35:21.815932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.815915 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:21.818968 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.818948 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-035bd5-dockercfg-8jmkr\"" Sep 10 16:35:21.819111 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.818999 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-035bd5\"" Sep 10 16:35:21.828281 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.828254 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp"] Sep 10 16:35:21.873126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.865986 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm"] Sep 10 16:35:21.876649 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.876624 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm"] Sep 10 16:35:21.876820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.876711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:21.922252 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.919960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2f4ea0-0d5b-4979-a488-b05b363f6835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp\" (UID: \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:21.922252 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.920062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fq9\" (UniqueName: \"kubernetes.io/projected/ec2f4ea0-0d5b-4979-a488-b05b363f6835-kube-api-access-77fq9\") pod \"payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp\" (UID: \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:21.955126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.954062 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:35:21.955126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:21.954154 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 16:35:22.020914 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.020881 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqrl7\" (UniqueName: \"kubernetes.io/projected/9dc111f8-3014-4e3d-933a-000c49e901c7-kube-api-access-mqrl7\") pod \"payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm\" (UID: \"9dc111f8-3014-4e3d-933a-000c49e901c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:22.020914 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.020920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2f4ea0-0d5b-4979-a488-b05b363f6835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp\" (UID: \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:22.021158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.020957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fq9\" (UniqueName: \"kubernetes.io/projected/ec2f4ea0-0d5b-4979-a488-b05b363f6835-kube-api-access-77fq9\") pod \"payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp\" (UID: \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:22.021158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.021073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc111f8-3014-4e3d-933a-000c49e901c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm\" (UID: \"9dc111f8-3014-4e3d-933a-000c49e901c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:22.021686 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.021667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2f4ea0-0d5b-4979-a488-b05b363f6835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp\" (UID: \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:22.029240 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.029217 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fq9\" (UniqueName: \"kubernetes.io/projected/ec2f4ea0-0d5b-4979-a488-b05b363f6835-kube-api-access-77fq9\") pod \"payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp\" (UID: \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:22.069933 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.069822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:22.122657 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.122612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqrl7\" (UniqueName: \"kubernetes.io/projected/9dc111f8-3014-4e3d-933a-000c49e901c7-kube-api-access-mqrl7\") pod \"payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm\" (UID: \"9dc111f8-3014-4e3d-933a-000c49e901c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:22.122842 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.122727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc111f8-3014-4e3d-933a-000c49e901c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm\" (UID: \"9dc111f8-3014-4e3d-933a-000c49e901c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:22.123455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.123435 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc111f8-3014-4e3d-933a-000c49e901c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm\" (UID: \"9dc111f8-3014-4e3d-933a-000c49e901c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:22.128858 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.128839 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:22.131776 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.131748 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqrl7\" (UniqueName: \"kubernetes.io/projected/9dc111f8-3014-4e3d-933a-000c49e901c7-kube-api-access-mqrl7\") pod \"payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm\" (UID: \"9dc111f8-3014-4e3d-933a-000c49e901c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:22.179083 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.178960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:22.189640 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.189610 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:22.359117 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.358800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:22.437422 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.437396 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp"] Sep 10 16:35:22.438722 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:35:22.438699 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2f4ea0_0d5b_4979_a488_b05b363f6835.slice/crio-34c3ade9e6a3ff8d04baf74fcd40e089f8ec7e62719862b0e5098ad4fccfa27e WatchSource:0}: Error finding container 34c3ade9e6a3ff8d04baf74fcd40e089f8ec7e62719862b0e5098ad4fccfa27e: Status 404 returned error can't find the container with id 34c3ade9e6a3ff8d04baf74fcd40e089f8ec7e62719862b0e5098ad4fccfa27e Sep 10 16:35:22.496233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.496206 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm"] Sep 10 16:35:22.497511 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:35:22.497482 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc111f8_3014_4e3d_933a_000c49e901c7.slice/crio-a6ed3d316fedaea24cbda3e2f71d4223574a07e58a22c30fd2ae39eabf6b47f4 WatchSource:0}: Error finding container a6ed3d316fedaea24cbda3e2f71d4223574a07e58a22c30fd2ae39eabf6b47f4: Status 404 returned error can't find the container with id a6ed3d316fedaea24cbda3e2f71d4223574a07e58a22c30fd2ae39eabf6b47f4 Sep 10 16:35:22.697006 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.696912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" event={"ID":"9dc111f8-3014-4e3d-933a-000c49e901c7","Type":"ContainerStarted","Data":"b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18"} Sep 10 16:35:22.697006 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.696976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" event={"ID":"9dc111f8-3014-4e3d-933a-000c49e901c7","Type":"ContainerStarted","Data":"a6ed3d316fedaea24cbda3e2f71d4223574a07e58a22c30fd2ae39eabf6b47f4"} Sep 10 16:35:22.697006 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.697008 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:22.699434 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.699407 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" event={"ID":"ec2f4ea0-0d5b-4979-a488-b05b363f6835","Type":"ContainerStarted","Data":"a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8"} Sep 10 16:35:22.699571 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.699457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" event={"ID":"ec2f4ea0-0d5b-4979-a488-b05b363f6835","Type":"ContainerStarted","Data":"34c3ade9e6a3ff8d04baf74fcd40e089f8ec7e62719862b0e5098ad4fccfa27e"} Sep 10 16:35:22.699571 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.699497 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:22.715677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.715609 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" podStartSLOduration=1.715594699 podStartE2EDuration="1.715594699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:35:22.713452234 +0000 UTC m=+1073.439771164" watchObservedRunningTime="2026-09-10 16:35:22.715594699 +0000 UTC m=+1073.441913626" Sep 10 16:35:22.731319 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.731245 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" podStartSLOduration=1.73122871 podStartE2EDuration="1.73122871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:35:22.729964419 +0000 UTC m=+1073.456283354" watchObservedRunningTime="2026-09-10 16:35:22.73122871 +0000 UTC m=+1073.457547639" Sep 10 16:35:22.954811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.954710 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 16:35:22.955094 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:22.955059 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 16:35:23.264160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:23.264056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:23.568096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:23.567984 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:23.957958 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:23.957871 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:35:23.958277 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:23.958256 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:35:23.958932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:23.958914 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:35:26.942846 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:26.942806 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:35:29.199313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:29.199267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:29.257712 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:29.257613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:30.394851 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:30.394816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:33.711945 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:33.711914 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:35:33.712404 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:33.711985 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:35:37.825508 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:37.825466 2562 generic.go:356] "Generic (PLEG): container finished" podID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerID="9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8" exitCode=0 Sep 10 16:35:37.825937 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:37.825521 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" event={"ID":"c57edd1e-1d1a-479a-8d8a-657e9cfc434d","Type":"ContainerDied","Data":"9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8"} Sep 10 16:35:37.825989 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:37.825974 2562 scope.go:117] "RemoveContainer" containerID="9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8" Sep 10 16:35:37.827194 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:37.827163 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:37.827259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:37.827237 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:37.827293 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:37.827273 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:35:38.833029 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:38.832994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" event={"ID":"c57edd1e-1d1a-479a-8d8a-657e9cfc434d","Type":"ContainerStarted","Data":"0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e"} Sep 10 16:35:39.291928 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:39.291889 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:39.292127 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:39.292096 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 16:35:39.292188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:39.292139 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 16:35:40.292188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:40.292146 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 16:35:40.292584 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:40.292209 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 16:35:41.251705 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:41.251642 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:35:41.292533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:41.292494 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 16:35:41.292929 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:41.292560 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 16:35:42.292134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:42.292090 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 16:35:42.292322 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:42.292167 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 16:35:43.292828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:43.292788 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 16:35:43.293251 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:43.292850 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 16:35:43.412429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:43.412397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:44.296644 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:44.296608 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:44.297152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:44.297010 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:44.298142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:44.298122 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:35:44.316061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:44.316002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:44.554188 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:44.554088 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:44.991949 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:44.991834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:45.697144 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:45.697109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:46.372936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:46.372899 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:53.881094 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:53.881060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:54.229578 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:54.229485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:54.450735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:54.450704 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:56.893228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:56.893189 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:57.548100 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:57.548065 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:57.646404 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:57.646374 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:58.473708 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:58.473676 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:59.303960 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:59.303922 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:35:59.932689 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:35:59.932656 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:00.710495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:00.710458 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:00.787091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:00.787024 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:00.889779 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:00.889746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:01.208359 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.208324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:01.304257 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.304224 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:01.360366 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.360331 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t"] Sep 10 16:36:01.363880 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.363863 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:01.366840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.366812 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8438b933-dockercfg-pbpgp\"" Sep 10 16:36:01.366977 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.366826 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8438b933\"" Sep 10 16:36:01.374488 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.374464 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t"] Sep 10 16:36:01.451283 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.451245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9sbq\" (UniqueName: \"kubernetes.io/projected/d994bf40-a893-4c41-80ef-f58517b895e1-kube-api-access-z9sbq\") pod \"payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t\" (UID: \"d994bf40-a893-4c41-80ef-f58517b895e1\") " pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:01.451845 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.451819 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d994bf40-a893-4c41-80ef-f58517b895e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t\" (UID: \"d994bf40-a893-4c41-80ef-f58517b895e1\") " pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:01.553135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.553095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d994bf40-a893-4c41-80ef-f58517b895e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t\" (UID: \"d994bf40-a893-4c41-80ef-f58517b895e1\") " pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:01.553325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.553155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9sbq\" (UniqueName: \"kubernetes.io/projected/d994bf40-a893-4c41-80ef-f58517b895e1-kube-api-access-z9sbq\") pod \"payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t\" (UID: \"d994bf40-a893-4c41-80ef-f58517b895e1\") " pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:01.553790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.553765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d994bf40-a893-4c41-80ef-f58517b895e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t\" (UID: \"d994bf40-a893-4c41-80ef-f58517b895e1\") " pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:01.561942 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.561910 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9sbq\" (UniqueName: \"kubernetes.io/projected/d994bf40-a893-4c41-80ef-f58517b895e1-kube-api-access-z9sbq\") pod \"payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t\" (UID: \"d994bf40-a893-4c41-80ef-f58517b895e1\") " pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:01.675837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.675799 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:01.953254 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.953223 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t"] Sep 10 16:36:01.955530 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:36:01.955504 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd994bf40_a893_4c41_80ef_f58517b895e1.slice/crio-418939373ae02abe1518401964bc25999b4effb6a3338e7631083aedabcde7a9 WatchSource:0}: Error finding container 418939373ae02abe1518401964bc25999b4effb6a3338e7631083aedabcde7a9: Status 404 returned error can't find the container with id 418939373ae02abe1518401964bc25999b4effb6a3338e7631083aedabcde7a9 Sep 10 16:36:01.993102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:01.993069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" event={"ID":"d994bf40-a893-4c41-80ef-f58517b895e1","Type":"ContainerStarted","Data":"418939373ae02abe1518401964bc25999b4effb6a3338e7631083aedabcde7a9"} Sep 10 16:36:03.000702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:03.000663 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" event={"ID":"d994bf40-a893-4c41-80ef-f58517b895e1","Type":"ContainerStarted","Data":"8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6"} Sep 10 16:36:03.001142 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:03.000818 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:03.022894 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:03.022835 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" podStartSLOduration=2.022814316 podStartE2EDuration="2.022814316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:36:03.021944738 +0000 UTC m=+1113.748263668" watchObservedRunningTime="2026-09-10 16:36:03.022814316 +0000 UTC m=+1113.749133246" Sep 10 16:36:08.143530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:08.143493 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:36:14.008708 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:14.008677 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:36:22.564454 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:22.564421 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:23.002698 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:23.002666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:23.865793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:23.865759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:24.946199 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:24.946166 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:25.692723 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:25.692690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:26.563549 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:26.563518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:27.301247 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:27.301217 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:27.726781 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:27.726687 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:29.066923 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:29.066885 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:36:29.907890 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:29.907845 2562 scope.go:117] "RemoveContainer" containerID="ce5c89c7e2da07bf4745d731fc3cd147e98b2ae69a30feb9cce17c82e7b1384e" Sep 10 16:36:29.917102 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:29.917078 2562 scope.go:117] "RemoveContainer" containerID="7e141378d90b88719e9c5c265e9d59cf8d536078ab624d426c23857e7d952170" Sep 10 16:36:29.925654 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:29.925629 2562 scope.go:117] "RemoveContainer" containerID="d78c789e21bb4331eebf9444c18a1169a0768260124688fc80c10472cab6b10a" Sep 10 16:36:40.040233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:40.040194 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:40.114549 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:40.114512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:43.550204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:43.550172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:36:49.465048 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:49.464999 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:36:51.691908 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:36:51.691871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:03.341680 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:03.341637 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:37:06.985256 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:06.985221 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w"] Sep 10 16:37:06.987828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:06.987810 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" Sep 10 16:37:06.991546 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:06.991516 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:37:06.991692 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:06.991564 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:37:07.000555 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:07.000523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w"] Sep 10 16:37:07.144192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:07.144146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736f2b8e-9243-4f26-8606-e0f4d172e4fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8438b933-88m2w\" (UID: \"736f2b8e-9243-4f26-8606-e0f4d172e4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" Sep 10 16:37:07.245771 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:07.245739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736f2b8e-9243-4f26-8606-e0f4d172e4fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8438b933-88m2w\" (UID: \"736f2b8e-9243-4f26-8606-e0f4d172e4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" Sep 10 16:37:07.246480 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:07.246457 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736f2b8e-9243-4f26-8606-e0f4d172e4fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8438b933-88m2w\" (UID: \"736f2b8e-9243-4f26-8606-e0f4d172e4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" Sep 10 16:37:07.277413 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:07.277377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:07.298445 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:07.298408 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" Sep 10 16:37:07.575889 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:07.575864 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w"] Sep 10 16:37:07.578123 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:37:07.578089 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736f2b8e_9243_4f26_8606_e0f4d172e4fe.slice/crio-fa441dd22a4b2f36c91ffc4192942d2def0724264370194bf828d3d04ba7669f WatchSource:0}: Error finding container fa441dd22a4b2f36c91ffc4192942d2def0724264370194bf828d3d04ba7669f: Status 404 returned error can't find the container with id fa441dd22a4b2f36c91ffc4192942d2def0724264370194bf828d3d04ba7669f Sep 10 16:37:08.077077 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:08.077018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:08.376544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:08.376510 2562 generic.go:356] "Generic (PLEG): container finished" podID="736f2b8e-9243-4f26-8606-e0f4d172e4fe" containerID="aee0058374640bf0927a30bcdd865ae4579e10e6971cb4aba6e08ef7b6929fff" exitCode=0 Sep 10 16:37:08.376723 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:08.376569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" event={"ID":"736f2b8e-9243-4f26-8606-e0f4d172e4fe","Type":"ContainerDied","Data":"aee0058374640bf0927a30bcdd865ae4579e10e6971cb4aba6e08ef7b6929fff"} Sep 10 16:37:08.376723 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:08.376615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" event={"ID":"736f2b8e-9243-4f26-8606-e0f4d172e4fe","Type":"ContainerStarted","Data":"fa441dd22a4b2f36c91ffc4192942d2def0724264370194bf828d3d04ba7669f"} Sep 10 16:37:08.912276 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:08.912242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:09.516769 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:09.516745 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" Sep 10 16:37:09.674676 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:09.674581 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736f2b8e-9243-4f26-8606-e0f4d172e4fe-maas-api-service-ca\") pod \"736f2b8e-9243-4f26-8606-e0f4d172e4fe\" (UID: \"736f2b8e-9243-4f26-8606-e0f4d172e4fe\") " Sep 10 16:37:09.675070 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:09.675027 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736f2b8e-9243-4f26-8606-e0f4d172e4fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "736f2b8e-9243-4f26-8606-e0f4d172e4fe" (UID: "736f2b8e-9243-4f26-8606-e0f4d172e4fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:37:09.683418 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:09.683389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:09.775743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:09.775710 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/736f2b8e-9243-4f26-8606-e0f4d172e4fe-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:37:10.388732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:10.388704 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" Sep 10 16:37:10.388732 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:10.388716 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w" event={"ID":"736f2b8e-9243-4f26-8606-e0f4d172e4fe","Type":"ContainerDied","Data":"fa441dd22a4b2f36c91ffc4192942d2def0724264370194bf828d3d04ba7669f"} Sep 10 16:37:10.388940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:10.388759 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa441dd22a4b2f36c91ffc4192942d2def0724264370194bf828d3d04ba7669f" Sep 10 16:37:10.660959 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:10.660868 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:11.449859 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:11.449822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:12.412998 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:12.412960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:21.865451 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:21.865411 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:37:22.488853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.488812 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:22.520767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.520729 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t"] Sep 10 16:37:22.521098 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.521023 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" podUID="d994bf40-a893-4c41-80ef-f58517b895e1" containerName="payload-processing" containerID="cri-o://8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6" gracePeriod=30 Sep 10 16:37:22.696506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.696469 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w"] Sep 10 16:37:22.698668 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.698641 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8438b933-88m2w"] Sep 10 16:37:22.793163 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.793132 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:37:22.913669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.913631 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d994bf40-a893-4c41-80ef-f58517b895e1-plugins-config-volume\") pod \"d994bf40-a893-4c41-80ef-f58517b895e1\" (UID: \"d994bf40-a893-4c41-80ef-f58517b895e1\") " Sep 10 16:37:22.914090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.913730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9sbq\" (UniqueName: \"kubernetes.io/projected/d994bf40-a893-4c41-80ef-f58517b895e1-kube-api-access-z9sbq\") pod \"d994bf40-a893-4c41-80ef-f58517b895e1\" (UID: \"d994bf40-a893-4c41-80ef-f58517b895e1\") " Sep 10 16:37:22.914090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.914012 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d994bf40-a893-4c41-80ef-f58517b895e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d994bf40-a893-4c41-80ef-f58517b895e1" (UID: "d994bf40-a893-4c41-80ef-f58517b895e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:37:22.915836 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:22.915813 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d994bf40-a893-4c41-80ef-f58517b895e1-kube-api-access-z9sbq" (OuterVolumeSpecName: "kube-api-access-z9sbq") pod "d994bf40-a893-4c41-80ef-f58517b895e1" (UID: "d994bf40-a893-4c41-80ef-f58517b895e1"). InnerVolumeSpecName "kube-api-access-z9sbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:37:23.015500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.015458 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d994bf40-a893-4c41-80ef-f58517b895e1-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:37:23.015500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.015492 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9sbq\" (UniqueName: \"kubernetes.io/projected/d994bf40-a893-4c41-80ef-f58517b895e1-kube-api-access-z9sbq\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:37:23.468784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.468744 2562 generic.go:356] "Generic (PLEG): container finished" podID="d994bf40-a893-4c41-80ef-f58517b895e1" containerID="8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6" exitCode=0 Sep 10 16:37:23.468976 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.468791 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" Sep 10 16:37:23.468976 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.468815 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" event={"ID":"d994bf40-a893-4c41-80ef-f58517b895e1","Type":"ContainerDied","Data":"8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6"} Sep 10 16:37:23.468976 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.468883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t" event={"ID":"d994bf40-a893-4c41-80ef-f58517b895e1","Type":"ContainerDied","Data":"418939373ae02abe1518401964bc25999b4effb6a3338e7631083aedabcde7a9"} Sep 10 16:37:23.468976 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.468906 2562 scope.go:117] "RemoveContainer" containerID="8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6" Sep 10 16:37:23.478932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.478903 2562 scope.go:117] "RemoveContainer" containerID="8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6" Sep 10 16:37:23.479277 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:37:23.479257 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6\": container with ID starting with 8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6 not found: ID does not exist" containerID="8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6" Sep 10 16:37:23.479335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.479291 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6"} err="failed to get container status \"8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6\": rpc error: code = NotFound desc = could not find container \"8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6\": container with ID starting with 8dd7d6f9d398a46a106cac920e338d35fb908d57f182be2b2b38c015598d62d6 not found: ID does not exist" Sep 10 16:37:23.498448 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.498416 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t"] Sep 10 16:37:23.507505 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.507473 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8438b933-d74b579d5-5sg4t"] Sep 10 16:37:23.758233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.758203 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="736f2b8e-9243-4f26-8606-e0f4d172e4fe" path="/var/lib/kubelet/pods/736f2b8e-9243-4f26-8606-e0f4d172e4fe/volumes" Sep 10 16:37:23.758590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.758575 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d994bf40-a893-4c41-80ef-f58517b895e1" path="/var/lib/kubelet/pods/d994bf40-a893-4c41-80ef-f58517b895e1/volumes" Sep 10 16:37:23.995307 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:23.995263 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:24.093083 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:24.092966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:24.757316 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:24.757285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:25.704938 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:25.704901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:25.816882 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:25.816851 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:26.458398 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:26.458355 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:29.783913 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:29.783882 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:37:29.785146 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:29.785122 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:37:29.955843 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:29.955807 2562 scope.go:117] "RemoveContainer" containerID="b1d58f487d49b04d4d46ad7ef1b31507f0924b41fee6dc4d16a169c358185837" Sep 10 16:37:29.964812 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:29.964790 2562 scope.go:117] "RemoveContainer" containerID="b15db04fa21cf6694dc1c0faacbdf8da3abab4ca0db3277cf775aebc05bd374f" Sep 10 16:37:30.883542 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:30.883506 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:30.983306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:30.983274 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:37:39.054228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:39.054141 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:37:53.837385 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:37:53.837348 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:38:08.068831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:08.068793 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:08.185533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:08.185498 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:08.939532 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:08.939499 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:38:14.020209 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:14.020169 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:23.322136 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:23.322030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:23.422846 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:23.422803 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:26.651587 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:26.651547 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:38:29.993824 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:29.993786 2562 scope.go:117] "RemoveContainer" containerID="ee3ff8b8d91b372b1bb62e676cfea6dd320166b6766f7e8144e5609c62486ffa" Sep 10 16:38:30.002505 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:30.002480 2562 scope.go:117] "RemoveContainer" containerID="270c7f935832721cd2f4d2fccf2c1026d39910a29a9a67c27978786c57c88848" Sep 10 16:38:32.651501 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:32.651466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:33.565138 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:33.565099 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:37.953836 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:37.953801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:38.053726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:38.053696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:38.168442 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:38.168403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:38.437015 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:38.436976 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:38:39.403001 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:39.402963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:39.501809 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:39.501773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:39.619360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:39.619330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:40.203456 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:40.203424 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:40.308352 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:40.308318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:41.638014 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:41.637981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:47.712234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:47.712190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:47.812249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:47.812208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:48.465263 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:48.465230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:48.548837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:48.548801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:52.454754 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:52.454718 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:38:54.473853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:54.473818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:54.535892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:54.535814 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:54.653289 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:54.653248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:38:54.758368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:38:54.758327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:05.209149 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:05.209115 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:05.429911 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:05.429878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:10.805075 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:10.805019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:15.049816 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:15.049778 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:39:26.360281 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:26.360250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:30.026667 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:30.026633 2562 scope.go:117] "RemoveContainer" containerID="e80d50fbd8e4b05a6dd348dd218780deb6f35226fd574302b2e9352f4c1fc01b" Sep 10 16:39:31.545229 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:31.545190 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:39:33.402544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:33.402510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:33.499217 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:33.499185 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:33.761051 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:33.760997 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:34.089787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:34.089701 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:34.214087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:34.214019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:36.383452 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:36.383403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:37.012828 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:37.012780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:37.110420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:37.110390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:37.410681 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:37.410574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:38.110399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:38.110366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:38.964236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:38.964201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:39.697633 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:39.697598 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:40.397306 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:40.397270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:40.475553 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:40.475519 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:40.618437 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:40.618394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:40.992371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:40.992337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:41.089322 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:41.089289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:50.056540 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:50.056505 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:39:52.601347 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:52.601265 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:39:58.966473 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:39:58.966439 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:00.173850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:00.173818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:01.422221 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:01.422173 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:03.028909 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:03.028848 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:03.916260 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:03.916221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:04.906655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:04.906586 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:05.702612 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:05.702575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:06.432499 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:06.432466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:07.067467 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.067404 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:07.307643 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.307607 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:07.541263 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.541227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:07.636815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.636787 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:07.840358 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.840272 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4"] Sep 10 16:40:07.840970 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.840950 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d994bf40-a893-4c41-80ef-f58517b895e1" containerName="payload-processing" Sep 10 16:40:07.840970 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.840971 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d994bf40-a893-4c41-80ef-f58517b895e1" containerName="payload-processing" Sep 10 16:40:07.841093 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.840994 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="736f2b8e-9243-4f26-8606-e0f4d172e4fe" containerName="revoke-keys" Sep 10 16:40:07.841093 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.840999 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="736f2b8e-9243-4f26-8606-e0f4d172e4fe" containerName="revoke-keys" Sep 10 16:40:07.841158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.841098 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="736f2b8e-9243-4f26-8606-e0f4d172e4fe" containerName="revoke-keys" Sep 10 16:40:07.841158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.841107 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d994bf40-a893-4c41-80ef-f58517b895e1" containerName="payload-processing" Sep 10 16:40:07.844186 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.844169 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.848121 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.848089 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cdfd80-openshift-default-dockercfg-vchd6\"" Sep 10 16:40:07.857248 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.857219 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4"] Sep 10 16:40:07.875605 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.875571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:40:07.882954 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.882924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:07.909616 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909578 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-certs\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.909787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-data\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.909787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-podinfo\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.909787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cfzw\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-kube-api-access-9cfzw\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.909787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istiod-ca-cert\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.909787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909777 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-socket\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.909961 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-envoy\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.909961 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-credential-socket\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:07.909961 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:07.909954 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-token\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.010669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-credential-socket\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.010884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-token\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.010884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-certs\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.010884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-data\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.010884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-podinfo\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.010884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cfzw\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-kube-api-access-9cfzw\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.011210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istiod-ca-cert\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.011210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-socket\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.011210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.010997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-envoy\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.011210 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.011163 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-credential-socket\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.011428 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.011244 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-certs\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.011428 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.011377 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-data\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.011536 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.011470 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-socket\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.011827 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.011803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istiod-ca-cert\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.013262 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.013241 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-envoy\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.013523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.013506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-podinfo\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.019916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.019893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-token\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.020000 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.019963 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cfzw\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-kube-api-access-9cfzw\") pod \"e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.035724 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.035685 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:08.157240 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.157143 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:08.448415 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.448265 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4"] Sep 10 16:40:08.454622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.454563 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:40:08.454865 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.454840 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:40:08.455072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:08.455015 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:40:09.391293 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:09.391248 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" event={"ID":"cc53d131-d72f-43f6-bf8f-bd063b12e8c4","Type":"ContainerStarted","Data":"ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11"} Sep 10 16:40:09.391807 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:09.391308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" event={"ID":"cc53d131-d72f-43f6-bf8f-bd063b12e8c4","Type":"ContainerStarted","Data":"64e7963b8400a043bc357e937331db542fe14c1afe67f84218d533563c320175"} Sep 10 16:40:09.431849 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:09.431793 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" podStartSLOduration=2.431774901 podStartE2EDuration="2.431774901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:40:09.431445468 +0000 UTC m=+1360.157764421" watchObservedRunningTime="2026-09-10 16:40:09.431774901 +0000 UTC m=+1360.158093830" Sep 10 16:40:10.158299 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:10.158263 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:10.163711 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:10.163675 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:10.396337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:10.396304 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:10.397332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:10.397312 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:10.794940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:10.794910 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:11.432428 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:11.432395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:11.629124 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:11.629091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:11.880575 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:11.880510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:13.087004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:13.086972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:14.003767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:14.003731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:15.425987 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:15.425953 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:16.421391 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:16.421358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:17.427124 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:17.427085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:17.553889 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:17.553837 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:17.660356 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:17.660320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:18.016133 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.016098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:18.113842 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.113809 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:18.200963 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.200928 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8"] Sep 10 16:40:18.204715 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.204696 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:18.208361 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.208338 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cdfd80-dockercfg-8mrkx\"" Sep 10 16:40:18.208502 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.208338 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cdfd80\"" Sep 10 16:40:18.216134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.216107 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8"] Sep 10 16:40:18.327679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.327588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vq9\" (UniqueName: \"kubernetes.io/projected/ca7c89a2-6a00-4428-8c55-024b6f394c95-kube-api-access-q8vq9\") pod \"payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8\" (UID: \"ca7c89a2-6a00-4428-8c55-024b6f394c95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:18.327679 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.327665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7c89a2-6a00-4428-8c55-024b6f394c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8\" (UID: \"ca7c89a2-6a00-4428-8c55-024b6f394c95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:18.428973 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.428931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vq9\" (UniqueName: \"kubernetes.io/projected/ca7c89a2-6a00-4428-8c55-024b6f394c95-kube-api-access-q8vq9\") pod \"payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8\" (UID: \"ca7c89a2-6a00-4428-8c55-024b6f394c95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:18.429460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.429000 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7c89a2-6a00-4428-8c55-024b6f394c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8\" (UID: \"ca7c89a2-6a00-4428-8c55-024b6f394c95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:18.429715 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.429691 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7c89a2-6a00-4428-8c55-024b6f394c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8\" (UID: \"ca7c89a2-6a00-4428-8c55-024b6f394c95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:18.441752 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.441719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vq9\" (UniqueName: \"kubernetes.io/projected/ca7c89a2-6a00-4428-8c55-024b6f394c95-kube-api-access-q8vq9\") pod \"payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8\" (UID: \"ca7c89a2-6a00-4428-8c55-024b6f394c95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:18.517203 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.517162 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:18.523375 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.523345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:18.797485 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:18.797449 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8"] Sep 10 16:40:18.800530 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:40:18.800494 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7c89a2_6a00_4428_8c55_024b6f394c95.slice/crio-b0474755be90f63c029f2d1e897b852ab817ea72519f315359750358f6c2775b WatchSource:0}: Error finding container b0474755be90f63c029f2d1e897b852ab817ea72519f315359750358f6c2775b: Status 404 returned error can't find the container with id b0474755be90f63c029f2d1e897b852ab817ea72519f315359750358f6c2775b Sep 10 16:40:19.358278 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:19.358244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:19.450026 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:19.449986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" event={"ID":"ca7c89a2-6a00-4428-8c55-024b6f394c95","Type":"ContainerStarted","Data":"7aff27975074b7e592897aa49dbe876070db58acfa1734180dc01199374db4ba"} Sep 10 16:40:19.450504 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:19.450055 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" event={"ID":"ca7c89a2-6a00-4428-8c55-024b6f394c95","Type":"ContainerStarted","Data":"b0474755be90f63c029f2d1e897b852ab817ea72519f315359750358f6c2775b"} Sep 10 16:40:19.450504 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:19.450182 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:19.472252 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:19.472195 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" podStartSLOduration=1.472176844 podStartE2EDuration="1.472176844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:40:19.470289525 +0000 UTC m=+1370.196608455" watchObservedRunningTime="2026-09-10 16:40:19.472176844 +0000 UTC m=+1370.198495774" Sep 10 16:40:20.323333 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:20.323292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:23.769706 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:23.769666 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:40:30.052661 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:30.052619 2562 scope.go:117] "RemoveContainer" containerID="881a0ed430a0991a685505343e1a57e511dacb5b1171335fa2dd37e342c88814" Sep 10 16:40:30.458847 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:30.458743 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:40:34.538694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:34.538659 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerID="ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11" exitCode=0 Sep 10 16:40:34.539236 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:34.538738 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" event={"ID":"cc53d131-d72f-43f6-bf8f-bd063b12e8c4","Type":"ContainerDied","Data":"ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11"} Sep 10 16:40:34.539371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:34.539352 2562 scope.go:117] "RemoveContainer" containerID="ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11" Sep 10 16:40:34.540574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:34.540543 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:40:34.540661 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:34.540615 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:40:34.540661 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:34.540647 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:40:35.549164 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:35.549118 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" event={"ID":"cc53d131-d72f-43f6-bf8f-bd063b12e8c4","Type":"ContainerStarted","Data":"7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39"} Sep 10 16:40:36.157888 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:36.157787 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:36.159365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:36.158981 2562 patch_prober.go:28] interesting pod/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:40:36.159365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:36.159053 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:40:37.157739 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:37.157697 2562 patch_prober.go:28] interesting pod/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:40:37.158361 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:37.158324 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:40:38.158672 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:38.158622 2562 patch_prober.go:28] interesting pod/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:40:38.159202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:38.158698 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:40:38.525512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:38.525480 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:39.157925 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:39.157869 2562 patch_prober.go:28] interesting pod/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:40:39.158173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:39.157942 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:40:39.934656 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:39.934615 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:40.157791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:40.157754 2562 patch_prober.go:28] interesting pod/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:40:40.158133 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:40.158102 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:40:40.396646 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:40.396599 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:40.827921 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:40.827884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:41.162312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:41.162233 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:41.163297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:41.163274 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:40:41.497289 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:41.497244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:41.915727 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:41.915631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:42.678169 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:42.678136 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:43.393195 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:43.393160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:44.454513 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:44.454484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:45.354482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:45.354440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:46.122833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:46.122794 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:47.030289 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:47.030252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:47.882875 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:47.882838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:48.777512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:48.777470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:49.725271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:49.725237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:50.747518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:50.747481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:52.593098 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:52.593059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:54.558641 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:54.558601 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:40:58.202171 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:40:58.202136 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:41:00.286951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:00.286908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:00.411369 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:00.411337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:00.560605 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:00.560511 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:00.746352 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:00.746310 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:01.109847 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:01.109816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:01.276857 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:01.276820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:02.316230 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:02.316198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:03.162703 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:03.162664 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:04.316146 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:04.316098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:05.523340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:05.523297 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:06.552413 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:06.552379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:06.682801 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:06.682766 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:10.162347 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:10.162312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:10.455677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:10.455571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:10.674804 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:10.674764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:11.248735 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:11.248583 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:12.843568 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:12.843523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:14.098420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:14.098380 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:15.374021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:15.373984 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:15.697458 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:15.697361 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:41:16.442394 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:16.442358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:17.427280 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:17.427244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:18.358314 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:18.358280 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:26.648645 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:26.648557 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:26.706675 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:26.706628 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8"] Sep 10 16:41:26.706922 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:26.706893 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" podUID="ca7c89a2-6a00-4428-8c55-024b6f394c95" containerName="payload-pre-processing" containerID="cri-o://7aff27975074b7e592897aa49dbe876070db58acfa1734180dc01199374db4ba" gracePeriod=30 Sep 10 16:41:26.749405 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:26.749374 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:26.844349 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:26.844313 2562 generic.go:356] "Generic (PLEG): container finished" podID="ca7c89a2-6a00-4428-8c55-024b6f394c95" containerID="7aff27975074b7e592897aa49dbe876070db58acfa1734180dc01199374db4ba" exitCode=0 Sep 10 16:41:26.844534 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:26.844386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" event={"ID":"ca7c89a2-6a00-4428-8c55-024b6f394c95","Type":"ContainerDied","Data":"7aff27975074b7e592897aa49dbe876070db58acfa1734180dc01199374db4ba"} Sep 10 16:41:26.856929 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:26.856898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:26.982522 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:26.982448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:27.008311 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.008279 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:41:27.074332 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.074290 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7c89a2-6a00-4428-8c55-024b6f394c95-plugins-config-volume\") pod \"ca7c89a2-6a00-4428-8c55-024b6f394c95\" (UID: \"ca7c89a2-6a00-4428-8c55-024b6f394c95\") " Sep 10 16:41:27.074532 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.074359 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8vq9\" (UniqueName: \"kubernetes.io/projected/ca7c89a2-6a00-4428-8c55-024b6f394c95-kube-api-access-q8vq9\") pod \"ca7c89a2-6a00-4428-8c55-024b6f394c95\" (UID: \"ca7c89a2-6a00-4428-8c55-024b6f394c95\") " Sep 10 16:41:27.074678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.074651 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7c89a2-6a00-4428-8c55-024b6f394c95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca7c89a2-6a00-4428-8c55-024b6f394c95" (UID: "ca7c89a2-6a00-4428-8c55-024b6f394c95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:41:27.076460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.076435 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca7c89a2-6a00-4428-8c55-024b6f394c95-kube-api-access-q8vq9" (OuterVolumeSpecName: "kube-api-access-q8vq9") pod "ca7c89a2-6a00-4428-8c55-024b6f394c95" (UID: "ca7c89a2-6a00-4428-8c55-024b6f394c95"). InnerVolumeSpecName "kube-api-access-q8vq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:41:27.175462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.175414 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7c89a2-6a00-4428-8c55-024b6f394c95-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:27.175462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.175454 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8vq9\" (UniqueName: \"kubernetes.io/projected/ca7c89a2-6a00-4428-8c55-024b6f394c95-kube-api-access-q8vq9\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:27.323150 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.323116 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:27.456144 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.456107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:27.851475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.851441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" event={"ID":"ca7c89a2-6a00-4428-8c55-024b6f394c95","Type":"ContainerDied","Data":"b0474755be90f63c029f2d1e897b852ab817ea72519f315359750358f6c2775b"} Sep 10 16:41:27.851475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.851473 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8" Sep 10 16:41:27.851983 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.851514 2562 scope.go:117] "RemoveContainer" containerID="7aff27975074b7e592897aa49dbe876070db58acfa1734180dc01199374db4ba" Sep 10 16:41:27.874000 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.873817 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8"] Sep 10 16:41:27.876279 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:27.876255 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cdfd80-684cf85898-97tv8"] Sep 10 16:41:28.284437 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:28.284390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:29.593822 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:29.593768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:29.758904 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:29.758870 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca7c89a2-6a00-4428-8c55-024b6f394c95" path="/var/lib/kubelet/pods/ca7c89a2-6a00-4428-8c55-024b6f394c95/volumes" Sep 10 16:41:30.684602 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:30.684570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:31.524087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:31.522172 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:41:31.937529 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:31.937432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:35.256770 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:35.256736 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:35.364265 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:35.364222 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:35.537665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:35.537580 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4"] Sep 10 16:41:35.537819 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:35.537782 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" containerID="cri-o://7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39" gracePeriod=30 Sep 10 16:41:35.548730 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:35.548698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:36.193950 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:36.193912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:37.052928 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:37.052891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:37.313500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:37.313407 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:37.826943 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:37.826909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:38.249195 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:38.249163 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:38.603161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:38.603068 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:38.719063 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:38.719012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:39.085510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:39.085475 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:39.179689 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:39.179656 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:39.467228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:39.467144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:40.362702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.362658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:40.794941 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.794914 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:41:40.926706 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926603 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-data\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.926706 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926700 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cfzw\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-kube-api-access-9cfzw\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.926951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926764 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-podinfo\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.926951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926800 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-credential-socket\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.926951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926816 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-token\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.926951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926863 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-envoy\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.926951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926891 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istiod-ca-cert\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.926951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926915 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-socket\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.926951 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926919 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:40.927364 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.926958 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-certs\") pod \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\" (UID: \"cc53d131-d72f-43f6-bf8f-bd063b12e8c4\") " Sep 10 16:41:40.927364 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.927326 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:41:40.927506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.927467 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:40.927506 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.927489 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:40.927656 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.927627 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:40.927782 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.927750 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:40.928015 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.927991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:40.929688 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.929667 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-kube-api-access-9cfzw" (OuterVolumeSpecName: "kube-api-access-9cfzw") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "kube-api-access-9cfzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:41:40.929832 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.929696 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:41:40.929943 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.929898 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:41:40.930072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.929992 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc53d131-d72f-43f6-bf8f-bd063b12e8c4" (UID: "cc53d131-d72f-43f6-bf8f-bd063b12e8c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:40.935323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.935295 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerID="7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39" exitCode=0 Sep 10 16:41:40.935449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.935344 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" Sep 10 16:41:40.935449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.935355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" event={"ID":"cc53d131-d72f-43f6-bf8f-bd063b12e8c4","Type":"ContainerDied","Data":"7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39"} Sep 10 16:41:40.935449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.935393 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4" event={"ID":"cc53d131-d72f-43f6-bf8f-bd063b12e8c4","Type":"ContainerDied","Data":"64e7963b8400a043bc357e937331db542fe14c1afe67f84218d533563c320175"} Sep 10 16:41:40.935449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.935416 2562 scope.go:117] "RemoveContainer" containerID="7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39" Sep 10 16:41:40.952914 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.952892 2562 scope.go:117] "RemoveContainer" containerID="ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11" Sep 10 16:41:40.961151 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.961132 2562 scope.go:117] "RemoveContainer" containerID="7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39" Sep 10 16:41:40.961416 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:41:40.961396 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39\": container with ID starting with 7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39 not found: ID does not exist" containerID="7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39" Sep 10 16:41:40.961470 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.961427 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39"} err="failed to get container status \"7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39\": rpc error: code = NotFound desc = could not find container \"7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39\": container with ID starting with 7b5050930546323bb2d4570a7fb74d009d6d91276d17e67b3fec2e97652a6e39 not found: ID does not exist" Sep 10 16:41:40.961470 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.961445 2562 scope.go:117] "RemoveContainer" containerID="ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11" Sep 10 16:41:40.961675 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:41:40.961659 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11\": container with ID starting with ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11 not found: ID does not exist" containerID="ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11" Sep 10 16:41:40.961716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.961680 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11"} err="failed to get container status \"ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11\": rpc error: code = NotFound desc = could not find container \"ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11\": container with ID starting with ba438660f0cca71a49ed752edc7ee8a0da0a2439bb50723146a8379028908b11 not found: ID does not exist" Sep 10 16:41:40.965866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.965835 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4"] Sep 10 16:41:40.970845 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:40.970820 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cdfd80-openshift-default-74c55b8d56-vnqm4"] Sep 10 16:41:41.028125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.028086 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:41.028125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.028118 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:41.028125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.028130 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:41.028380 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.028145 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cfzw\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-kube-api-access-9cfzw\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:41.028380 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.028158 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:41.028380 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.028169 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:41.028380 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.028179 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc53d131-d72f-43f6-bf8f-bd063b12e8c4-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:41.290113 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.290082 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:41.757930 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.757895 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" path="/var/lib/kubelet/pods/cc53d131-d72f-43f6-bf8f-bd063b12e8c4/volumes" Sep 10 16:41:41.969303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:41.969270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:42.091906 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:42.091827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:42.189309 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:42.189261 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:42.931994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:42.931950 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:44.092637 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:44.092596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:44.909249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:44.909213 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:45.707590 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:45.707558 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:45.812229 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:45.812193 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:46.086281 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.086248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:46.189523 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.189480 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:46.286241 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.286193 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:46.390213 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.390131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:46.395805 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.395767 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr"] Sep 10 16:41:46.396326 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.396308 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" Sep 10 16:41:46.396326 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.396327 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" Sep 10 16:41:46.396457 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.396354 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca7c89a2-6a00-4428-8c55-024b6f394c95" containerName="payload-pre-processing" Sep 10 16:41:46.396457 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.396361 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca7c89a2-6a00-4428-8c55-024b6f394c95" containerName="payload-pre-processing" Sep 10 16:41:46.399716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.397192 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" Sep 10 16:41:46.399716 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.397236 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca7c89a2-6a00-4428-8c55-024b6f394c95" containerName="payload-pre-processing" Sep 10 16:41:46.406869 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.406843 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh"] Sep 10 16:41:46.406987 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.406965 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:46.407368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.407352 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" Sep 10 16:41:46.407368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.407367 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" Sep 10 16:41:46.407524 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.407447 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc53d131-d72f-43f6-bf8f-bd063b12e8c4" containerName="istio-proxy" Sep 10 16:41:46.410303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.410280 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95d261\"" Sep 10 16:41:46.410401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.410313 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-95d261-dockercfg-zhfg8\"" Sep 10 16:41:46.411449 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.411433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:46.420024 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.419996 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr"] Sep 10 16:41:46.422271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.422250 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh"] Sep 10 16:41:46.491193 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.489997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f7290-bee5-4fd1-b383-d2acc5f8f000-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh\" (UID: \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:46.491193 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.490094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wz5w\" (UniqueName: \"kubernetes.io/projected/553a8c3d-2573-4812-86f7-a059ad681ca2-kube-api-access-2wz5w\") pod \"payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr\" (UID: \"553a8c3d-2573-4812-86f7-a059ad681ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:46.491193 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.490140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553a8c3d-2573-4812-86f7-a059ad681ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr\" (UID: \"553a8c3d-2573-4812-86f7-a059ad681ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:46.491193 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.490232 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5p82\" (UniqueName: \"kubernetes.io/projected/a47f7290-bee5-4fd1-b383-d2acc5f8f000-kube-api-access-r5p82\") pod \"payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh\" (UID: \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:46.591152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.591104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f7290-bee5-4fd1-b383-d2acc5f8f000-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh\" (UID: \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:46.591358 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.591250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wz5w\" (UniqueName: \"kubernetes.io/projected/553a8c3d-2573-4812-86f7-a059ad681ca2-kube-api-access-2wz5w\") pod \"payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr\" (UID: \"553a8c3d-2573-4812-86f7-a059ad681ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:46.591358 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.591279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553a8c3d-2573-4812-86f7-a059ad681ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr\" (UID: \"553a8c3d-2573-4812-86f7-a059ad681ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:46.591358 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.591344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5p82\" (UniqueName: \"kubernetes.io/projected/a47f7290-bee5-4fd1-b383-d2acc5f8f000-kube-api-access-r5p82\") pod \"payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh\" (UID: \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:46.591977 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.591949 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f7290-bee5-4fd1-b383-d2acc5f8f000-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh\" (UID: \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:46.592107 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.592076 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553a8c3d-2573-4812-86f7-a059ad681ca2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr\" (UID: \"553a8c3d-2573-4812-86f7-a059ad681ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:46.603609 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.603576 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:46.603773 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.603725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wz5w\" (UniqueName: \"kubernetes.io/projected/553a8c3d-2573-4812-86f7-a059ad681ca2-kube-api-access-2wz5w\") pod \"payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr\" (UID: \"553a8c3d-2573-4812-86f7-a059ad681ca2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:46.604090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.604068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5p82\" (UniqueName: \"kubernetes.io/projected/a47f7290-bee5-4fd1-b383-d2acc5f8f000-kube-api-access-r5p82\") pod \"payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh\" (UID: \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:46.703790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.703715 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:46.720813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.720783 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:46.729096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.729065 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:46.808619 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.808588 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:46.888359 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:46.888322 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:47.014553 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:47.014524 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr"] Sep 10 16:41:47.017185 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:41:47.017150 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553a8c3d_2573_4812_86f7_a059ad681ca2.slice/crio-f05ab4c7ca154a5e0672addba32a6a4baec42fe0725b17b96db129aa1c9ad736 WatchSource:0}: Error finding container f05ab4c7ca154a5e0672addba32a6a4baec42fe0725b17b96db129aa1c9ad736: Status 404 returned error can't find the container with id f05ab4c7ca154a5e0672addba32a6a4baec42fe0725b17b96db129aa1c9ad736 Sep 10 16:41:47.023114 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:47.023092 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh"] Sep 10 16:41:47.025700 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:41:47.025672 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47f7290_bee5_4fd1_b383_d2acc5f8f000.slice/crio-b895bd37db4488bc476f3942746babf77b9a61c7c0610c5cc046435be30a4300 WatchSource:0}: Error finding container b895bd37db4488bc476f3942746babf77b9a61c7c0610c5cc046435be30a4300: Status 404 returned error can't find the container with id b895bd37db4488bc476f3942746babf77b9a61c7c0610c5cc046435be30a4300 Sep 10 16:41:47.222075 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:47.222026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:47.593069 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:47.593009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:48.003287 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.003249 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" event={"ID":"553a8c3d-2573-4812-86f7-a059ad681ca2","Type":"ContainerStarted","Data":"22b0a223ee0eba00baf6e8bb470ca81430da049beaa99022bf9f62f7409cb3a2"} Sep 10 16:41:48.003750 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.003305 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" event={"ID":"553a8c3d-2573-4812-86f7-a059ad681ca2","Type":"ContainerStarted","Data":"f05ab4c7ca154a5e0672addba32a6a4baec42fe0725b17b96db129aa1c9ad736"} Sep 10 16:41:48.003750 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.003334 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:48.004905 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.004882 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" event={"ID":"a47f7290-bee5-4fd1-b383-d2acc5f8f000","Type":"ContainerStarted","Data":"9372b1ddecab116c443d2b1b2899dd309f50c32df9999f0405aa97ba285b0f6a"} Sep 10 16:41:48.005020 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.004922 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" event={"ID":"a47f7290-bee5-4fd1-b383-d2acc5f8f000","Type":"ContainerStarted","Data":"b895bd37db4488bc476f3942746babf77b9a61c7c0610c5cc046435be30a4300"} Sep 10 16:41:48.005020 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.004943 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:48.026382 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.026331 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" podStartSLOduration=2.026314287 podStartE2EDuration="2.026314287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:41:48.024077418 +0000 UTC m=+1458.750396373" watchObservedRunningTime="2026-09-10 16:41:48.026314287 +0000 UTC m=+1458.752633261" Sep 10 16:41:48.045538 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.045467 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" podStartSLOduration=2.04544366 podStartE2EDuration="2.04544366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:41:48.043341868 +0000 UTC m=+1458.769660812" watchObservedRunningTime="2026-09-10 16:41:48.04544366 +0000 UTC m=+1458.771762594" Sep 10 16:41:48.477098 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:48.476954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:49.291340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:49.291307 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:49.997808 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:49.997764 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:41:51.200319 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:51.200282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:51.512977 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:51.512947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:52.472885 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:52.472851 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:54.049565 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:54.049531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:54.165353 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:54.165321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:54.325019 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:54.324930 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg"] Sep 10 16:41:54.325204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:54.325154 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" containerID="cri-o://aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff" gracePeriod=30 Sep 10 16:41:54.349478 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:54.349442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:54.846287 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:54.846253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:55.062478 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:55.062429 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:55.913595 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:55.913561 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:57.038074 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:57.038015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:58.131678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:58.131642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:59.013394 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.013361 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:41:59.013542 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.013433 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:41:59.264251 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.264144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:41:59.577682 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.577653 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:41:59.738803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.738765 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-certs\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739003 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.738830 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-token\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739003 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.738884 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-envoy\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739003 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.738904 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75rcc\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-kube-api-access-75rcc\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739032 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-credential-socket\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739107 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-podinfo\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739149 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-data\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739167 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-socket\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739192 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istiod-ca-cert\") pod \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\" (UID: \"1c08a94d-1dd0-45c1-8821-2d6b930c0e95\") " Sep 10 16:41:59.739223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739189 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:59.739518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739433 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:59.739518 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739448 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:59.739632 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739531 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:41:59.739688 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739658 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:59.739852 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739834 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:59.739916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739856 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:59.739916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739869 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:59.739916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739883 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:59.739916 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.739896 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:59.741297 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.741272 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:41:59.741384 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.741356 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:41:59.741620 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.741599 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-kube-api-access-75rcc" (OuterVolumeSpecName: "kube-api-access-75rcc") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "kube-api-access-75rcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:41:59.741686 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.741626 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c08a94d-1dd0-45c1-8821-2d6b930c0e95" (UID: "1c08a94d-1dd0-45c1-8821-2d6b930c0e95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:41:59.841535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.841427 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:59.841535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.841465 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:59.841535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.841479 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rcc\" (UniqueName: \"kubernetes.io/projected/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-kube-api-access-75rcc\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:41:59.841535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:41:59.841492 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c08a94d-1dd0-45c1-8821-2d6b930c0e95-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:00.071910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.071875 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerID="aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff" exitCode=0 Sep 10 16:42:00.072091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.071930 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" Sep 10 16:42:00.072091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.071971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" event={"ID":"1c08a94d-1dd0-45c1-8821-2d6b930c0e95","Type":"ContainerDied","Data":"aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff"} Sep 10 16:42:00.072091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.072028 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg" event={"ID":"1c08a94d-1dd0-45c1-8821-2d6b930c0e95","Type":"ContainerDied","Data":"68a2fe37fbac851dd26c4b5399855afee060e40ccd16d45df9690c9cb13c03b0"} Sep 10 16:42:00.072091 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.072077 2562 scope.go:117] "RemoveContainer" containerID="aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff" Sep 10 16:42:00.081629 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.081604 2562 scope.go:117] "RemoveContainer" containerID="aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc" Sep 10 16:42:00.089932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.089915 2562 scope.go:117] "RemoveContainer" containerID="aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff" Sep 10 16:42:00.090241 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:42:00.090221 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff\": container with ID starting with aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff not found: ID does not exist" containerID="aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff" Sep 10 16:42:00.090296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.090251 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff"} err="failed to get container status \"aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff\": rpc error: code = NotFound desc = could not find container \"aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff\": container with ID starting with aae1abb524406f4566661149ce62c518e7b4d81773e0f5f021897cff09a642ff not found: ID does not exist" Sep 10 16:42:00.090296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.090267 2562 scope.go:117] "RemoveContainer" containerID="aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc" Sep 10 16:42:00.090517 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:42:00.090498 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc\": container with ID starting with aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc not found: ID does not exist" containerID="aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc" Sep 10 16:42:00.090574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.090526 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc"} err="failed to get container status \"aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc\": rpc error: code = NotFound desc = could not find container \"aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc\": container with ID starting with aa01a13aa507c7069ae6507e7699b2e629a36b697cfbd6b6d049f480261c41bc not found: ID does not exist" Sep 10 16:42:00.108800 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.108734 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg"] Sep 10 16:42:00.120913 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.120878 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bd1ece-openshift-default-7788c65cfd-wcwwg"] Sep 10 16:42:00.157891 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:00.157850 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:01.102130 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:01.102089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:01.757368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:01.757329 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" path="/var/lib/kubelet/pods/1c08a94d-1dd0-45c1-8821-2d6b930c0e95/volumes" Sep 10 16:42:01.824994 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:01.824948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:01.928086 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:01.928001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:02.517924 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:02.517879 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:02.840343 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:02.840262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:03.050095 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:03.050059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:04.858022 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:04.857989 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:05.601764 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.601725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:05.636441 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.636405 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm"] Sep 10 16:42:05.636905 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.636869 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" podUID="9dc111f8-3014-4e3d-933a-000c49e901c7" containerName="payload-processing" containerID="cri-o://b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18" gracePeriod=30 Sep 10 16:42:05.643767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.643346 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp"] Sep 10 16:42:05.643767 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.643588 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" podUID="ec2f4ea0-0d5b-4979-a488-b05b363f6835" containerName="payload-pre-processing" containerID="cri-o://a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8" gracePeriod=30 Sep 10 16:42:05.695585 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.695545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:05.767895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.767866 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:05.864635 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.864513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:05.945806 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.945767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:05.953581 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.953554 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:42:05.956979 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:05.956956 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:42:06.108867 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.108826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc111f8-3014-4e3d-933a-000c49e901c7-plugins-config-volume\") pod \"9dc111f8-3014-4e3d-933a-000c49e901c7\" (UID: \"9dc111f8-3014-4e3d-933a-000c49e901c7\") " Sep 10 16:42:06.109085 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.108878 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77fq9\" (UniqueName: \"kubernetes.io/projected/ec2f4ea0-0d5b-4979-a488-b05b363f6835-kube-api-access-77fq9\") pod \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\" (UID: \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\") " Sep 10 16:42:06.109085 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.108929 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2f4ea0-0d5b-4979-a488-b05b363f6835-plugins-config-volume\") pod \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\" (UID: \"ec2f4ea0-0d5b-4979-a488-b05b363f6835\") " Sep 10 16:42:06.109085 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.108973 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqrl7\" (UniqueName: \"kubernetes.io/projected/9dc111f8-3014-4e3d-933a-000c49e901c7-kube-api-access-mqrl7\") pod \"9dc111f8-3014-4e3d-933a-000c49e901c7\" (UID: \"9dc111f8-3014-4e3d-933a-000c49e901c7\") " Sep 10 16:42:06.109310 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.109282 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc111f8-3014-4e3d-933a-000c49e901c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9dc111f8-3014-4e3d-933a-000c49e901c7" (UID: "9dc111f8-3014-4e3d-933a-000c49e901c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:42:06.109355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.109311 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2f4ea0-0d5b-4979-a488-b05b363f6835-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec2f4ea0-0d5b-4979-a488-b05b363f6835" (UID: "ec2f4ea0-0d5b-4979-a488-b05b363f6835"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:42:06.109420 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.109405 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc111f8-3014-4e3d-933a-000c49e901c7-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:06.109459 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.109423 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2f4ea0-0d5b-4979-a488-b05b363f6835-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:06.111189 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.111161 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc111f8-3014-4e3d-933a-000c49e901c7-kube-api-access-mqrl7" (OuterVolumeSpecName: "kube-api-access-mqrl7") pod "9dc111f8-3014-4e3d-933a-000c49e901c7" (UID: "9dc111f8-3014-4e3d-933a-000c49e901c7"). InnerVolumeSpecName "kube-api-access-mqrl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:42:06.111298 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.111218 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2f4ea0-0d5b-4979-a488-b05b363f6835-kube-api-access-77fq9" (OuterVolumeSpecName: "kube-api-access-77fq9") pod "ec2f4ea0-0d5b-4979-a488-b05b363f6835" (UID: "ec2f4ea0-0d5b-4979-a488-b05b363f6835"). InnerVolumeSpecName "kube-api-access-77fq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:42:06.116535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.116455 2562 generic.go:356] "Generic (PLEG): container finished" podID="ec2f4ea0-0d5b-4979-a488-b05b363f6835" containerID="a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8" exitCode=0 Sep 10 16:42:06.116535 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.116496 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" Sep 10 16:42:06.116696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.116540 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" event={"ID":"ec2f4ea0-0d5b-4979-a488-b05b363f6835","Type":"ContainerDied","Data":"a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8"} Sep 10 16:42:06.116696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.116600 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp" event={"ID":"ec2f4ea0-0d5b-4979-a488-b05b363f6835","Type":"ContainerDied","Data":"34c3ade9e6a3ff8d04baf74fcd40e089f8ec7e62719862b0e5098ad4fccfa27e"} Sep 10 16:42:06.116696 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.116630 2562 scope.go:117] "RemoveContainer" containerID="a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8" Sep 10 16:42:06.118006 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.117982 2562 generic.go:356] "Generic (PLEG): container finished" podID="9dc111f8-3014-4e3d-933a-000c49e901c7" containerID="b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18" exitCode=0 Sep 10 16:42:06.118145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.118026 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" Sep 10 16:42:06.118145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.118030 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" event={"ID":"9dc111f8-3014-4e3d-933a-000c49e901c7","Type":"ContainerDied","Data":"b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18"} Sep 10 16:42:06.118145 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.118099 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm" event={"ID":"9dc111f8-3014-4e3d-933a-000c49e901c7","Type":"ContainerDied","Data":"a6ed3d316fedaea24cbda3e2f71d4223574a07e58a22c30fd2ae39eabf6b47f4"} Sep 10 16:42:06.127174 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.127156 2562 scope.go:117] "RemoveContainer" containerID="a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8" Sep 10 16:42:06.127464 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:42:06.127439 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8\": container with ID starting with a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8 not found: ID does not exist" containerID="a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8" Sep 10 16:42:06.127530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.127467 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8"} err="failed to get container status \"a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8\": rpc error: code = NotFound desc = could not find container \"a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8\": container with ID starting with a7c8741a6f807ed3562d2fa9cf2ce1201294919dd45e5feff646ffe49a2080e8 not found: ID does not exist" Sep 10 16:42:06.127530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.127484 2562 scope.go:117] "RemoveContainer" containerID="b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18" Sep 10 16:42:06.136203 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.136184 2562 scope.go:117] "RemoveContainer" containerID="b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18" Sep 10 16:42:06.136533 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:42:06.136513 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18\": container with ID starting with b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18 not found: ID does not exist" containerID="b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18" Sep 10 16:42:06.136636 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.136542 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18"} err="failed to get container status \"b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18\": rpc error: code = NotFound desc = could not find container \"b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18\": container with ID starting with b5233f9e026a8c8872cacfd0ddb0de9cf09e641bb43eb58e55c2087cf1d68c18 not found: ID does not exist" Sep 10 16:42:06.143547 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.143518 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm"] Sep 10 16:42:06.152328 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.152279 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-035bd5-84c7665477-vgnbm"] Sep 10 16:42:06.165129 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.165093 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp"] Sep 10 16:42:06.171786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.171753 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-035bd5-5648dbfbfd-92xpp"] Sep 10 16:42:06.186228 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.186199 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:06.210581 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.210544 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77fq9\" (UniqueName: \"kubernetes.io/projected/ec2f4ea0-0d5b-4979-a488-b05b363f6835-kube-api-access-77fq9\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:06.210581 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.210579 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqrl7\" (UniqueName: \"kubernetes.io/projected/9dc111f8-3014-4e3d-933a-000c49e901c7-kube-api-access-mqrl7\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:06.381090 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:06.380983 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:07.147851 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:07.147818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:07.757819 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:07.757777 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc111f8-3014-4e3d-933a-000c49e901c7" path="/var/lib/kubelet/pods/9dc111f8-3014-4e3d-933a-000c49e901c7/volumes" Sep 10 16:42:07.758326 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:07.758303 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2f4ea0-0d5b-4979-a488-b05b363f6835" path="/var/lib/kubelet/pods/ec2f4ea0-0d5b-4979-a488-b05b363f6835/volumes" Sep 10 16:42:08.183611 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:08.183523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:08.795202 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:08.795172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:08.979015 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:08.978970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:09.179490 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:09.179408 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:10.012352 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.012320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:10.104374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.104337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:10.525629 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.525597 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:10.631544 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.631508 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:10.656406 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.656363 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz"] Sep 10 16:42:10.657061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.656965 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc111f8-3014-4e3d-933a-000c49e901c7" containerName="payload-processing" Sep 10 16:42:10.657061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.656991 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc111f8-3014-4e3d-933a-000c49e901c7" containerName="payload-processing" Sep 10 16:42:10.657061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.657009 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" Sep 10 16:42:10.657061 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.657018 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" Sep 10 16:42:10.657360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.657088 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2f4ea0-0d5b-4979-a488-b05b363f6835" containerName="payload-pre-processing" Sep 10 16:42:10.657360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.657100 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2f4ea0-0d5b-4979-a488-b05b363f6835" containerName="payload-pre-processing" Sep 10 16:42:10.657360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.657221 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc111f8-3014-4e3d-933a-000c49e901c7" containerName="payload-processing" Sep 10 16:42:10.657360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.657236 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" Sep 10 16:42:10.657360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.657247 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" Sep 10 16:42:10.657360 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.657259 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2f4ea0-0d5b-4979-a488-b05b363f6835" containerName="payload-pre-processing" Sep 10 16:42:10.661851 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.661832 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:10.664945 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.664923 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86d230\"" Sep 10 16:42:10.665106 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.665062 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86d230-dockercfg-lxwlg\"" Sep 10 16:42:10.672371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.672348 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz"] Sep 10 16:42:10.760614 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.760567 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e276561-19a7-43a2-a262-8403e76ca550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz\" (UID: \"3e276561-19a7-43a2-a262-8403e76ca550\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:10.760791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.760692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm78h\" (UniqueName: \"kubernetes.io/projected/3e276561-19a7-43a2-a262-8403e76ca550-kube-api-access-dm78h\") pod \"payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz\" (UID: \"3e276561-19a7-43a2-a262-8403e76ca550\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:10.862437 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.862346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e276561-19a7-43a2-a262-8403e76ca550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz\" (UID: \"3e276561-19a7-43a2-a262-8403e76ca550\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:10.862437 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.862438 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm78h\" (UniqueName: \"kubernetes.io/projected/3e276561-19a7-43a2-a262-8403e76ca550-kube-api-access-dm78h\") pod \"payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz\" (UID: \"3e276561-19a7-43a2-a262-8403e76ca550\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:10.863193 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.863171 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e276561-19a7-43a2-a262-8403e76ca550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz\" (UID: \"3e276561-19a7-43a2-a262-8403e76ca550\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:10.873093 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.873065 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm78h\" (UniqueName: \"kubernetes.io/projected/3e276561-19a7-43a2-a262-8403e76ca550-kube-api-access-dm78h\") pod \"payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz\" (UID: \"3e276561-19a7-43a2-a262-8403e76ca550\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:10.973854 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:10.973817 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:11.076815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:11.076760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:11.245211 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:11.245120 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:42:11.248884 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:11.248859 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz"] Sep 10 16:42:11.251651 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:42:11.251626 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e276561_19a7_43a2_a262_8403e76ca550.slice/crio-4bd2aa5d2d4632908ce36c0f915d686c8306f68d11a8c274b41c29f883e320e7 WatchSource:0}: Error finding container 4bd2aa5d2d4632908ce36c0f915d686c8306f68d11a8c274b41c29f883e320e7: Status 404 returned error can't find the container with id 4bd2aa5d2d4632908ce36c0f915d686c8306f68d11a8c274b41c29f883e320e7 Sep 10 16:42:11.873478 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:11.873446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:12.163784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:12.163679 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" event={"ID":"3e276561-19a7-43a2-a262-8403e76ca550","Type":"ContainerStarted","Data":"91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8"} Sep 10 16:42:12.163784 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:12.163777 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" event={"ID":"3e276561-19a7-43a2-a262-8403e76ca550","Type":"ContainerStarted","Data":"4bd2aa5d2d4632908ce36c0f915d686c8306f68d11a8c274b41c29f883e320e7"} Sep 10 16:42:12.164205 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:12.163801 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:12.184822 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:12.184769 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" podStartSLOduration=2.184752138 podStartE2EDuration="2.184752138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:42:12.183629156 +0000 UTC m=+1482.909948099" watchObservedRunningTime="2026-09-10 16:42:12.184752138 +0000 UTC m=+1482.911071067" Sep 10 16:42:12.554703 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:12.554666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:13.203325 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:13.203289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:13.274534 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:13.274503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:13.447269 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:13.447229 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx"] Sep 10 16:42:13.447460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:13.447435 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" containerID="cri-o://0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e" gracePeriod=30 Sep 10 16:42:13.476426 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:13.476342 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:14.299017 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:14.298980 2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 16:42:14.299429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:14.299071 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:42:18.712129 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.712057 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:42:18.751487 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751457 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istiod-ca-cert\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.751653 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751496 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-envoy\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.751653 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751514 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-data\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.751653 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751614 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t9h5\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-kube-api-access-5t9h5\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.751838 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751678 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-credential-socket\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.751838 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751817 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-socket\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.751952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751837 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-data" (OuterVolumeSpecName: "istio-data") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:42:18.751952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751869 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-certs\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.751952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751900 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-podinfo\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.751952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751908 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:42:18.751952 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751945 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-token\") pod \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\" (UID: \"c57edd1e-1d1a-479a-8d8a-657e9cfc434d\") " Sep 10 16:42:18.752229 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.751981 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:42:18.752229 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.752077 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:42:18.752229 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.752112 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:42:18.752443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.752407 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:18.752443 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.752441 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:18.752627 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.752459 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:18.752627 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.752474 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:18.752627 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.752488 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:18.754076 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.754028 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:42:18.754189 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.754155 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-token" (OuterVolumeSpecName: "istio-token") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:42:18.754280 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.754261 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:42:18.754505 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.754488 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-kube-api-access-5t9h5" (OuterVolumeSpecName: "kube-api-access-5t9h5") pod "c57edd1e-1d1a-479a-8d8a-657e9cfc434d" (UID: "c57edd1e-1d1a-479a-8d8a-657e9cfc434d"). InnerVolumeSpecName "kube-api-access-5t9h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:42:18.853094 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.853024 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:18.853094 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.853088 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t9h5\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-kube-api-access-5t9h5\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:18.853094 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.853101 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:18.853336 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:18.853114 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57edd1e-1d1a-479a-8d8a-657e9cfc434d-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:42:19.210434 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.210397 2562 generic.go:356] "Generic (PLEG): container finished" podID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerID="0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e" exitCode=0 Sep 10 16:42:19.210612 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.210448 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" Sep 10 16:42:19.210612 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.210489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" event={"ID":"c57edd1e-1d1a-479a-8d8a-657e9cfc434d","Type":"ContainerDied","Data":"0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e"} Sep 10 16:42:19.210612 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.210549 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx" event={"ID":"c57edd1e-1d1a-479a-8d8a-657e9cfc434d","Type":"ContainerDied","Data":"bf075147563291e149c2cf09a4d54e3512c7ea4fca7623ac12c89e18ee8e83a6"} Sep 10 16:42:19.210612 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.210579 2562 scope.go:117] "RemoveContainer" containerID="0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e" Sep 10 16:42:19.220312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.220293 2562 scope.go:117] "RemoveContainer" containerID="9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8" Sep 10 16:42:19.228367 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.228348 2562 scope.go:117] "RemoveContainer" containerID="0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e" Sep 10 16:42:19.228679 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:42:19.228654 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e\": container with ID starting with 0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e not found: ID does not exist" containerID="0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e" Sep 10 16:42:19.228780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.228685 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e"} err="failed to get container status \"0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e\": rpc error: code = NotFound desc = could not find container \"0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e\": container with ID starting with 0b014c4c35aace07509f07f26ee08a69ec9ddebb06f43328a8a41af7f80a570e not found: ID does not exist" Sep 10 16:42:19.228780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.228704 2562 scope.go:117] "RemoveContainer" containerID="9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8" Sep 10 16:42:19.228954 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:42:19.228941 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8\": container with ID starting with 9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8 not found: ID does not exist" containerID="9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8" Sep 10 16:42:19.228995 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.228957 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8"} err="failed to get container status \"9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8\": rpc error: code = NotFound desc = could not find container \"9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8\": container with ID starting with 9f48ef51b9d080fd55d0b133e8f8d0d12264de6de0f03331309ab95e801ff8a8 not found: ID does not exist" Sep 10 16:42:19.238563 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.238534 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx"] Sep 10 16:42:19.243376 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.243349 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-035bd5-openshift-default-79f778c45d-xsvcx"] Sep 10 16:42:19.758614 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:19.758577 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" path="/var/lib/kubelet/pods/c57edd1e-1d1a-479a-8d8a-657e9cfc434d/volumes" Sep 10 16:42:22.753948 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:22.753912 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:42:23.171234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:23.171151 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:42:29.821789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:29.821755 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:42:29.823848 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:29.823826 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:42:32.084513 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:32.084472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:33.576270 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:33.576237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:33.674768 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:33.674731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:33.858312 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:33.858221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:34.684014 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:34.683980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:35.372923 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:35.372890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:42:38.863106 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:38.863072 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz_3e276561-19a7-43a2-a262-8403e76ca550/payload-processing/0.log" Sep 10 16:42:57.711946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:42:57.711868 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:01.657712 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:01.657671 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:43:05.475811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:05.475770 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:09.951139 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:09.951103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:10.027819 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:10.027780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:23.865713 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:23.865674 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:43:29.891605 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:29.891572 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh_a47f7290-bee5-4fd1-b383-d2acc5f8f000/payload-processing/0.log" Sep 10 16:43:30.119234 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.119201 2562 scope.go:117] "RemoveContainer" containerID="aee0058374640bf0927a30bcdd865ae4579e10e6971cb4aba6e08ef7b6929fff" Sep 10 16:43:30.150096 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.150009 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz_3e276561-19a7-43a2-a262-8403e76ca550/payload-processing/0.log" Sep 10 16:43:30.434303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.434220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:30.710622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.710521 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m"] Sep 10 16:43:30.711067 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.711032 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" Sep 10 16:43:30.711184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.711069 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c08a94d-1dd0-45c1-8821-2d6b930c0e95" containerName="istio-proxy" Sep 10 16:43:30.711184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.711095 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" Sep 10 16:43:30.711184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.711104 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" Sep 10 16:43:30.711184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.711137 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" Sep 10 16:43:30.711184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.711146 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" Sep 10 16:43:30.711442 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.711257 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" Sep 10 16:43:30.711442 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.711272 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57edd1e-1d1a-479a-8d8a-657e9cfc434d" containerName="istio-proxy" Sep 10 16:43:30.714502 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.714479 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.718961 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.718938 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-f9d55d-openshift-default-dockercfg-kq522\"" Sep 10 16:43:30.729567 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.729538 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m"] Sep 10 16:43:30.750940 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.750906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:30.779574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-envoy\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.779574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-token\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.779803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-podinfo\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.779803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42s5\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-kube-api-access-h42s5\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.779803 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-credential-socket\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.779903 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779843 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-socket\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.779903 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.779969 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-certs\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.780004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.779976 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-data\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881451 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-credential-socket\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-socket\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-certs\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-data\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-envoy\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-token\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-podinfo\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.881876 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42s5\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-kube-api-access-h42s5\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.882121 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-credential-socket\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.882121 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-socket\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.882121 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.881991 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-certs\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.882121 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.882083 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-data\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.882500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.882478 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.884015 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.883993 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-envoy\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.884499 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.884478 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-podinfo\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.887694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.887670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:30.891519 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.891497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42s5\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-kube-api-access-h42s5\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:30.891670 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:30.891651 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-token\") pod \"e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:31.025689 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:31.025647 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:31.305431 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:31.305076 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m"] Sep 10 16:43:31.308239 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:43:31.308206 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb36d077_4e83_40ca_af9f_8d52fd4828f9.slice/crio-fba7619ad4e51b44c5fa2dd2dfd08d1b19a88fac0787c9ad7487079071c3a879 WatchSource:0}: Error finding container fba7619ad4e51b44c5fa2dd2dfd08d1b19a88fac0787c9ad7487079071c3a879: Status 404 returned error can't find the container with id fba7619ad4e51b44c5fa2dd2dfd08d1b19a88fac0787c9ad7487079071c3a879 Sep 10 16:43:31.310819 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:31.310764 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:43:31.310914 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:31.310876 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:43:31.310957 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:31.310922 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:43:31.589422 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:31.589324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" event={"ID":"bb36d077-4e83-40ca-af9f-8d52fd4828f9","Type":"ContainerStarted","Data":"4e4c4921d2ffd831b1db20b7d7048439d14dbd66d38bf3a654fe23e7d5d31a47"} Sep 10 16:43:31.589422 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:31.589387 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" event={"ID":"bb36d077-4e83-40ca-af9f-8d52fd4828f9","Type":"ContainerStarted","Data":"fba7619ad4e51b44c5fa2dd2dfd08d1b19a88fac0787c9ad7487079071c3a879"} Sep 10 16:43:32.026762 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:32.026724 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:32.032336 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:32.032303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:32.057404 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:32.057349 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" podStartSLOduration=2.057330143 podStartE2EDuration="2.057330143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:43:31.615780467 +0000 UTC m=+1562.342099399" watchObservedRunningTime="2026-09-10 16:43:32.057330143 +0000 UTC m=+1562.783649074" Sep 10 16:43:32.596231 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:32.596200 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:32.597192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:32.597171 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:32.811934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:32.811894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:33.567577 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:33.567547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:33.675380 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:33.675341 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:33.768700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:33.768666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:33.868616 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:33.868532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:34.819692 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:34.819656 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:35.693521 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:35.693483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:36.376184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:36.376146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:37.052698 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.052652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:37.154691 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.154660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:37.384224 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.384132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:37.517230 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.517199 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:37.607936 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.607900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:37.662702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.662607 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g"] Sep 10 16:43:37.666520 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.666502 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:37.669483 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.669461 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f9d55d\"" Sep 10 16:43:37.669610 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.669554 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f9d55d-dockercfg-t9pj9\"" Sep 10 16:43:37.677414 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.677388 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g"] Sep 10 16:43:37.754873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.754833 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fknrq\" (UniqueName: \"kubernetes.io/projected/185138ee-62e4-4306-9074-6ff22b26c3f0-kube-api-access-fknrq\") pod \"payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g\" (UID: \"185138ee-62e4-4306-9074-6ff22b26c3f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:37.755097 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.754928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185138ee-62e4-4306-9074-6ff22b26c3f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g\" (UID: \"185138ee-62e4-4306-9074-6ff22b26c3f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:37.856371 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.856328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fknrq\" (UniqueName: \"kubernetes.io/projected/185138ee-62e4-4306-9074-6ff22b26c3f0-kube-api-access-fknrq\") pod \"payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g\" (UID: \"185138ee-62e4-4306-9074-6ff22b26c3f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:37.856559 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.856384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185138ee-62e4-4306-9074-6ff22b26c3f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g\" (UID: \"185138ee-62e4-4306-9074-6ff22b26c3f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:37.857117 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.857098 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185138ee-62e4-4306-9074-6ff22b26c3f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g\" (UID: \"185138ee-62e4-4306-9074-6ff22b26c3f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:37.868422 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.868401 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fknrq\" (UniqueName: \"kubernetes.io/projected/185138ee-62e4-4306-9074-6ff22b26c3f0-kube-api-access-fknrq\") pod \"payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g\" (UID: \"185138ee-62e4-4306-9074-6ff22b26c3f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:37.978683 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:37.978578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:38.252098 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:38.252073 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g"] Sep 10 16:43:38.254380 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:43:38.254353 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185138ee_62e4_4306_9074_6ff22b26c3f0.slice/crio-b15bbd4b9e00c21e732b916a58abac80e2f5c47940b35efc6d61578c686dba19 WatchSource:0}: Error finding container b15bbd4b9e00c21e732b916a58abac80e2f5c47940b35efc6d61578c686dba19: Status 404 returned error can't find the container with id b15bbd4b9e00c21e732b916a58abac80e2f5c47940b35efc6d61578c686dba19 Sep 10 16:43:38.632986 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:38.632890 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" event={"ID":"185138ee-62e4-4306-9074-6ff22b26c3f0","Type":"ContainerStarted","Data":"e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2"} Sep 10 16:43:38.632986 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:38.632949 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:38.632986 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:38.632975 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" event={"ID":"185138ee-62e4-4306-9074-6ff22b26c3f0","Type":"ContainerStarted","Data":"b15bbd4b9e00c21e732b916a58abac80e2f5c47940b35efc6d61578c686dba19"} Sep 10 16:43:38.654796 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:38.654736 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" podStartSLOduration=1.654718669 podStartE2EDuration="1.654718669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:43:38.654444792 +0000 UTC m=+1569.380763721" watchObservedRunningTime="2026-09-10 16:43:38.654718669 +0000 UTC m=+1569.381037600" Sep 10 16:43:42.646678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:42.646638 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:43:49.642262 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:49.642218 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:43:53.721909 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:53.721876 2562 generic.go:356] "Generic (PLEG): container finished" podID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerID="4e4c4921d2ffd831b1db20b7d7048439d14dbd66d38bf3a654fe23e7d5d31a47" exitCode=0 Sep 10 16:43:53.722365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:53.721948 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" event={"ID":"bb36d077-4e83-40ca-af9f-8d52fd4828f9","Type":"ContainerDied","Data":"4e4c4921d2ffd831b1db20b7d7048439d14dbd66d38bf3a654fe23e7d5d31a47"} Sep 10 16:43:53.722416 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:53.722399 2562 scope.go:117] "RemoveContainer" containerID="4e4c4921d2ffd831b1db20b7d7048439d14dbd66d38bf3a654fe23e7d5d31a47" Sep 10 16:43:53.723495 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:53.723469 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:43:53.723551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:53.723534 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:43:53.723587 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:53.723567 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 16:43:54.729919 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:54.729884 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" event={"ID":"bb36d077-4e83-40ca-af9f-8d52fd4828f9","Type":"ContainerStarted","Data":"f850e03af5cb3928ea63380667b5b6b17215c2ed20e24644cd14540e7a86777b"} Sep 10 16:43:55.026796 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:55.026761 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:43:55.027003 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:55.026979 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 10 16:43:55.027099 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:55.027077 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 10 16:43:56.027060 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:56.027000 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 10 16:43:56.027445 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:56.027137 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 10 16:43:57.026688 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:57.026651 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 10 16:43:57.026867 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:57.026712 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 10 16:43:58.026308 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:58.026269 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 10 16:43:58.026719 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:58.026342 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 10 16:43:58.958972 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:58.958935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:43:59.027027 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:59.026989 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 10 16:43:59.027519 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:59.027087 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 10 16:43:59.774248 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:43:59.774206 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:00.217655 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:00.217567 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:44:00.218840 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:00.218814 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:44:00.219954 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:00.219932 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:44:00.523423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:00.523386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:01.196450 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:01.196412 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:01.892100 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:01.892069 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:02.585607 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:02.585577 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:03.629123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:03.629091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:05.184064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:05.184009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:05.937323 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:05.937285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:06.598353 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:06.598321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:07.377273 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:07.377236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:08.047744 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:08.047702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:08.746761 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:08.746722 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:15.518860 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.518819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:15.547022 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.546985 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g"] Sep 10 16:44:15.547255 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.547229 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" podUID="185138ee-62e4-4306-9074-6ff22b26c3f0" containerName="payload-processing" containerID="cri-o://e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2" gracePeriod=30 Sep 10 16:44:15.621537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.621508 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:15.720541 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.720503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:15.807861 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.807836 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:44:15.817539 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.817503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:15.856724 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.856685 2562 generic.go:356] "Generic (PLEG): container finished" podID="185138ee-62e4-4306-9074-6ff22b26c3f0" containerID="e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2" exitCode=0 Sep 10 16:44:15.856905 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.856748 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" event={"ID":"185138ee-62e4-4306-9074-6ff22b26c3f0","Type":"ContainerDied","Data":"e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2"} Sep 10 16:44:15.856905 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.856819 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" event={"ID":"185138ee-62e4-4306-9074-6ff22b26c3f0","Type":"ContainerDied","Data":"b15bbd4b9e00c21e732b916a58abac80e2f5c47940b35efc6d61578c686dba19"} Sep 10 16:44:15.856905 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.856757 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g" Sep 10 16:44:15.857045 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.856855 2562 scope.go:117] "RemoveContainer" containerID="e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2" Sep 10 16:44:15.865946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.865924 2562 scope.go:117] "RemoveContainer" containerID="e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2" Sep 10 16:44:15.866273 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:44:15.866251 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2\": container with ID starting with e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2 not found: ID does not exist" containerID="e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2" Sep 10 16:44:15.866341 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.866283 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2"} err="failed to get container status \"e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2\": rpc error: code = NotFound desc = could not find container \"e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2\": container with ID starting with e3a8e17e63a718d4162412cbe1ae0f03682e8c1a0ebb07b45fafa98e5305d7b2 not found: ID does not exist" Sep 10 16:44:15.958934 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.958894 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fknrq\" (UniqueName: \"kubernetes.io/projected/185138ee-62e4-4306-9074-6ff22b26c3f0-kube-api-access-fknrq\") pod \"185138ee-62e4-4306-9074-6ff22b26c3f0\" (UID: \"185138ee-62e4-4306-9074-6ff22b26c3f0\") " Sep 10 16:44:15.959123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.959107 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185138ee-62e4-4306-9074-6ff22b26c3f0-plugins-config-volume\") pod \"185138ee-62e4-4306-9074-6ff22b26c3f0\" (UID: \"185138ee-62e4-4306-9074-6ff22b26c3f0\") " Sep 10 16:44:15.959488 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.959459 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185138ee-62e4-4306-9074-6ff22b26c3f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "185138ee-62e4-4306-9074-6ff22b26c3f0" (UID: "185138ee-62e4-4306-9074-6ff22b26c3f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:44:15.960971 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:15.960953 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185138ee-62e4-4306-9074-6ff22b26c3f0-kube-api-access-fknrq" (OuterVolumeSpecName: "kube-api-access-fknrq") pod "185138ee-62e4-4306-9074-6ff22b26c3f0" (UID: "185138ee-62e4-4306-9074-6ff22b26c3f0"). InnerVolumeSpecName "kube-api-access-fknrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:44:16.060530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:16.060442 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185138ee-62e4-4306-9074-6ff22b26c3f0-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:16.060530 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:16.060474 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fknrq\" (UniqueName: \"kubernetes.io/projected/185138ee-62e4-4306-9074-6ff22b26c3f0-kube-api-access-fknrq\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:16.084296 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:16.084260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:16.182442 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:16.182411 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g"] Sep 10 16:44:16.184827 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:16.184780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:16.186066 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:16.186022 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9d55d-68b86f8bd5-bmm5g"] Sep 10 16:44:16.863687 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:16.863653 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:17.622948 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:17.622909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:17.757947 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:17.757915 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="185138ee-62e4-4306-9074-6ff22b26c3f0" path="/var/lib/kubelet/pods/185138ee-62e4-4306-9074-6ff22b26c3f0/volumes" Sep 10 16:44:18.581472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:18.581421 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:19.296516 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:19.296481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:19.952694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:19.952654 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:44:22.449315 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:22.449233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:22.549425 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:22.549387 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:22.679606 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:22.679568 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m"] Sep 10 16:44:22.679850 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:22.679818 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" containerID="cri-o://f850e03af5cb3928ea63380667b5b6b17215c2ed20e24644cd14540e7a86777b" gracePeriod=30 Sep 10 16:44:22.706129 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:22.706028 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:26.583462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:26.583429 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:27.471331 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.471298 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:27.928995 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.928963 2562 generic.go:356] "Generic (PLEG): container finished" podID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerID="f850e03af5cb3928ea63380667b5b6b17215c2ed20e24644cd14540e7a86777b" exitCode=0 Sep 10 16:44:27.929399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.929027 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" event={"ID":"bb36d077-4e83-40ca-af9f-8d52fd4828f9","Type":"ContainerDied","Data":"f850e03af5cb3928ea63380667b5b6b17215c2ed20e24644cd14540e7a86777b"} Sep 10 16:44:27.929399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.929095 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" event={"ID":"bb36d077-4e83-40ca-af9f-8d52fd4828f9","Type":"ContainerDied","Data":"fba7619ad4e51b44c5fa2dd2dfd08d1b19a88fac0787c9ad7487079071c3a879"} Sep 10 16:44:27.929399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.929114 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fba7619ad4e51b44c5fa2dd2dfd08d1b19a88fac0787c9ad7487079071c3a879" Sep 10 16:44:27.929399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.929128 2562 scope.go:117] "RemoveContainer" containerID="4e4c4921d2ffd831b1db20b7d7048439d14dbd66d38bf3a654fe23e7d5d31a47" Sep 10 16:44:27.939510 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.939488 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:44:27.982674 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982638 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-podinfo\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.982866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982691 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istiod-ca-cert\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.982866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982728 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-socket\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.982866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982778 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h42s5\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-kube-api-access-h42s5\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.982866 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982817 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-credential-socket\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.983099 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982882 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-certs\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.983099 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982912 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-token\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.983099 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982939 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-data\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.983099 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.982969 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-envoy\") pod \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\" (UID: \"bb36d077-4e83-40ca-af9f-8d52fd4828f9\") " Sep 10 16:44:27.983313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.983102 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:44:27.983313 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.983239 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:44:27.983482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.983340 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:44:27.983482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.983381 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:44:27.983482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.983424 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:44:27.983482 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.983478 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:27.983684 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.983492 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:27.983684 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.983500 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:27.985135 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.985106 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:44:27.985472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.985447 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-kube-api-access-h42s5" (OuterVolumeSpecName: "kube-api-access-h42s5") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "kube-api-access-h42s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:44:27.985787 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.985769 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:44:27.985873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:27.985795 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb36d077-4e83-40ca-af9f-8d52fd4828f9" (UID: "bb36d077-4e83-40ca-af9f-8d52fd4828f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:44:28.084204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.084112 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42s5\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-kube-api-access-h42s5\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:28.084204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.084146 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:28.084204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.084157 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:28.084204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.084165 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:28.084204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.084174 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:28.084204 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.084181 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb36d077-4e83-40ca-af9f-8d52fd4828f9-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:28.161740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.161701 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:28.935729 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.935687 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m" Sep 10 16:44:28.963650 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.963617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:28.979162 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.979124 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m"] Sep 10 16:44:28.981118 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:28.981098 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f9d55d-openshift-default-b489b57d6-r4n4m"] Sep 10 16:44:29.688700 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:29.688661 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:29.759448 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:29.759417 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" path="/var/lib/kubelet/pods/bb36d077-4e83-40ca-af9f-8d52fd4828f9/volumes" Sep 10 16:44:30.391638 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:30.391607 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:39.464356 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:39.464316 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:44:41.806265 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:41.806231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:41.840499 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:41.840466 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh"] Sep 10 16:44:41.840718 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:41.840691 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" podUID="a47f7290-bee5-4fd1-b383-d2acc5f8f000" containerName="payload-processing" containerID="cri-o://9372b1ddecab116c443d2b1b2899dd309f50c32df9999f0405aa97ba285b0f6a" gracePeriod=30 Sep 10 16:44:41.848215 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:41.848190 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr"] Sep 10 16:44:41.848701 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:41.848671 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" podUID="553a8c3d-2573-4812-86f7-a059ad681ca2" containerName="payload-pre-processing" containerID="cri-o://22b0a223ee0eba00baf6e8bb470ca81430da049beaa99022bf9f62f7409cb3a2" gracePeriod=30 Sep 10 16:44:41.888226 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:41.888195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:41.961293 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:41.961246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:42.004642 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.004544 2562 generic.go:356] "Generic (PLEG): container finished" podID="553a8c3d-2573-4812-86f7-a059ad681ca2" containerID="22b0a223ee0eba00baf6e8bb470ca81430da049beaa99022bf9f62f7409cb3a2" exitCode=0 Sep 10 16:44:42.004642 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.004609 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" event={"ID":"553a8c3d-2573-4812-86f7-a059ad681ca2","Type":"ContainerDied","Data":"22b0a223ee0eba00baf6e8bb470ca81430da049beaa99022bf9f62f7409cb3a2"} Sep 10 16:44:42.006316 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.006290 2562 generic.go:356] "Generic (PLEG): container finished" podID="a47f7290-bee5-4fd1-b383-d2acc5f8f000" containerID="9372b1ddecab116c443d2b1b2899dd309f50c32df9999f0405aa97ba285b0f6a" exitCode=0 Sep 10 16:44:42.006432 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.006363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" event={"ID":"a47f7290-bee5-4fd1-b383-d2acc5f8f000","Type":"ContainerDied","Data":"9372b1ddecab116c443d2b1b2899dd309f50c32df9999f0405aa97ba285b0f6a"} Sep 10 16:44:42.034831 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.034804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:42.138783 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.138756 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:44:42.141853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.141836 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:44:42.222315 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.222280 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553a8c3d-2573-4812-86f7-a059ad681ca2-plugins-config-volume\") pod \"553a8c3d-2573-4812-86f7-a059ad681ca2\" (UID: \"553a8c3d-2573-4812-86f7-a059ad681ca2\") " Sep 10 16:44:42.222504 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.222326 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f7290-bee5-4fd1-b383-d2acc5f8f000-plugins-config-volume\") pod \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\" (UID: \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\") " Sep 10 16:44:42.222504 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.222447 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wz5w\" (UniqueName: \"kubernetes.io/projected/553a8c3d-2573-4812-86f7-a059ad681ca2-kube-api-access-2wz5w\") pod \"553a8c3d-2573-4812-86f7-a059ad681ca2\" (UID: \"553a8c3d-2573-4812-86f7-a059ad681ca2\") " Sep 10 16:44:42.222504 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.222487 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5p82\" (UniqueName: \"kubernetes.io/projected/a47f7290-bee5-4fd1-b383-d2acc5f8f000-kube-api-access-r5p82\") pod \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\" (UID: \"a47f7290-bee5-4fd1-b383-d2acc5f8f000\") " Sep 10 16:44:42.222718 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.222692 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553a8c3d-2573-4812-86f7-a059ad681ca2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "553a8c3d-2573-4812-86f7-a059ad681ca2" (UID: "553a8c3d-2573-4812-86f7-a059ad681ca2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:44:42.222802 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.222780 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47f7290-bee5-4fd1-b383-d2acc5f8f000-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a47f7290-bee5-4fd1-b383-d2acc5f8f000" (UID: "a47f7290-bee5-4fd1-b383-d2acc5f8f000"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:44:42.224625 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.224602 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47f7290-bee5-4fd1-b383-d2acc5f8f000-kube-api-access-r5p82" (OuterVolumeSpecName: "kube-api-access-r5p82") pod "a47f7290-bee5-4fd1-b383-d2acc5f8f000" (UID: "a47f7290-bee5-4fd1-b383-d2acc5f8f000"). InnerVolumeSpecName "kube-api-access-r5p82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:44:42.224734 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.224647 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553a8c3d-2573-4812-86f7-a059ad681ca2-kube-api-access-2wz5w" (OuterVolumeSpecName: "kube-api-access-2wz5w") pod "553a8c3d-2573-4812-86f7-a059ad681ca2" (UID: "553a8c3d-2573-4812-86f7-a059ad681ca2"). InnerVolumeSpecName "kube-api-access-2wz5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:44:42.324147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.324109 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wz5w\" (UniqueName: \"kubernetes.io/projected/553a8c3d-2573-4812-86f7-a059ad681ca2-kube-api-access-2wz5w\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:42.324147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.324141 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5p82\" (UniqueName: \"kubernetes.io/projected/a47f7290-bee5-4fd1-b383-d2acc5f8f000-kube-api-access-r5p82\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:42.324147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.324151 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553a8c3d-2573-4812-86f7-a059ad681ca2-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:42.324147 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.324160 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47f7290-bee5-4fd1-b383-d2acc5f8f000-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:44:42.393303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.393222 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:42.494870 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:42.494815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:43.013634 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.013602 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" Sep 10 16:44:43.013634 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.013616 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr" event={"ID":"553a8c3d-2573-4812-86f7-a059ad681ca2","Type":"ContainerDied","Data":"f05ab4c7ca154a5e0672addba32a6a4baec42fe0725b17b96db129aa1c9ad736"} Sep 10 16:44:43.014184 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.013681 2562 scope.go:117] "RemoveContainer" containerID="22b0a223ee0eba00baf6e8bb470ca81430da049beaa99022bf9f62f7409cb3a2" Sep 10 16:44:43.015123 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.015103 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" Sep 10 16:44:43.015213 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.015135 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh" event={"ID":"a47f7290-bee5-4fd1-b383-d2acc5f8f000","Type":"ContainerDied","Data":"b895bd37db4488bc476f3942746babf77b9a61c7c0610c5cc046435be30a4300"} Sep 10 16:44:43.023460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.023324 2562 scope.go:117] "RemoveContainer" containerID="9372b1ddecab116c443d2b1b2899dd309f50c32df9999f0405aa97ba285b0f6a" Sep 10 16:44:43.039933 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.039902 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr"] Sep 10 16:44:43.046157 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.046127 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95d261-798748886b-cgjwr"] Sep 10 16:44:43.063338 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.063306 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh"] Sep 10 16:44:43.066249 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.066225 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95d261-7c5f95cf99-5xzgh"] Sep 10 16:44:43.303379 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.303345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:43.764030 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.763992 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="553a8c3d-2573-4812-86f7-a059ad681ca2" path="/var/lib/kubelet/pods/553a8c3d-2573-4812-86f7-a059ad681ca2/volumes" Sep 10 16:44:43.764488 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:43.764468 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a47f7290-bee5-4fd1-b383-d2acc5f8f000" path="/var/lib/kubelet/pods/a47f7290-bee5-4fd1-b383-d2acc5f8f000/volumes" Sep 10 16:44:44.165346 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:44.165263 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:44.892945 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:44.892902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:47.495088 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:47.495055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:47.814394 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:47.814314 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:44:53.151103 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:53.151063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:44:53.647259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:44:53.647226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:00.148669 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.148613 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh"] Sep 10 16:45:00.149370 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149341 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="553a8c3d-2573-4812-86f7-a059ad681ca2" containerName="payload-pre-processing" Sep 10 16:45:00.149370 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149370 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="553a8c3d-2573-4812-86f7-a059ad681ca2" containerName="payload-pre-processing" Sep 10 16:45:00.149593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149391 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="185138ee-62e4-4306-9074-6ff22b26c3f0" containerName="payload-processing" Sep 10 16:45:00.149593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149400 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="185138ee-62e4-4306-9074-6ff22b26c3f0" containerName="payload-processing" Sep 10 16:45:00.149593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149412 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" Sep 10 16:45:00.149593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149420 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" Sep 10 16:45:00.149593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149461 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47f7290-bee5-4fd1-b383-d2acc5f8f000" containerName="payload-processing" Sep 10 16:45:00.149593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149469 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47f7290-bee5-4fd1-b383-d2acc5f8f000" containerName="payload-processing" Sep 10 16:45:00.149593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149575 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="553a8c3d-2573-4812-86f7-a059ad681ca2" containerName="payload-pre-processing" Sep 10 16:45:00.149593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149588 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" Sep 10 16:45:00.150004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149599 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="185138ee-62e4-4306-9074-6ff22b26c3f0" containerName="payload-processing" Sep 10 16:45:00.150004 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.149612 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a47f7290-bee5-4fd1-b383-d2acc5f8f000" containerName="payload-processing" Sep 10 16:45:00.154217 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.154192 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs"] Sep 10 16:45:00.154364 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.154314 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" Sep 10 16:45:00.154652 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.154633 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" Sep 10 16:45:00.154725 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.154655 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" Sep 10 16:45:00.154816 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.154803 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb36d077-4e83-40ca-af9f-8d52fd4828f9" containerName="istio-proxy" Sep 10 16:45:00.157581 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.157560 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:45:00.157797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.157779 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" Sep 10 16:45:00.165072 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.165021 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh"] Sep 10 16:45:00.166736 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.166716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" Sep 10 16:45:00.167843 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.167819 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs"] Sep 10 16:45:00.176001 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.175980 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" Sep 10 16:45:00.448551 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.448522 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh"] Sep 10 16:45:00.450121 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:45:00.450093 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2147dd83_38bf_4d08_993b_1419c302a873.slice/crio-549c31f3b158f119d24c34d34edccf2a7d0e55ba745e1c84c2f1b5240703d1d1 WatchSource:0}: Error finding container 549c31f3b158f119d24c34d34edccf2a7d0e55ba745e1c84c2f1b5240703d1d1: Status 404 returned error can't find the container with id 549c31f3b158f119d24c34d34edccf2a7d0e55ba745e1c84c2f1b5240703d1d1 Sep 10 16:45:00.468981 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:00.468959 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs"] Sep 10 16:45:00.472071 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:45:00.472019 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a186ee_9a9f_41ab_b903_b5d3c31b132b.slice/crio-d37fbdd8fa7b14e4910caabcd54047a3b7e1de27037a7c3c43049038190eea4c WatchSource:0}: Error finding container d37fbdd8fa7b14e4910caabcd54047a3b7e1de27037a7c3c43049038190eea4c: Status 404 returned error can't find the container with id d37fbdd8fa7b14e4910caabcd54047a3b7e1de27037a7c3c43049038190eea4c Sep 10 16:45:01.107467 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:01.107355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" event={"ID":"2147dd83-38bf-4d08-993b-1419c302a873","Type":"ContainerStarted","Data":"50a3999b35f515553a48457d90ddff2097ed3ee6e23d41a8ee32580be56e0401"} Sep 10 16:45:01.107640 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:01.107465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" event={"ID":"2147dd83-38bf-4d08-993b-1419c302a873","Type":"ContainerStarted","Data":"549c31f3b158f119d24c34d34edccf2a7d0e55ba745e1c84c2f1b5240703d1d1"} Sep 10 16:45:01.109206 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:01.109173 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" event={"ID":"12a186ee-9a9f-41ab-b903-b5d3c31b132b","Type":"ContainerStarted","Data":"a35069963dc2bfd3e2a5fc15c6f7aae46f77860e21f8bb04a4abfb80cb1bbc87"} Sep 10 16:45:01.109362 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:01.109224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" event={"ID":"12a186ee-9a9f-41ab-b903-b5d3c31b132b","Type":"ContainerStarted","Data":"d37fbdd8fa7b14e4910caabcd54047a3b7e1de27037a7c3c43049038190eea4c"} Sep 10 16:45:01.126404 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:01.126349 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" podStartSLOduration=1.1263335620000001 podStartE2EDuration="1.126333562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:45:01.123991238 +0000 UTC m=+1651.850310169" watchObservedRunningTime="2026-09-10 16:45:01.126333562 +0000 UTC m=+1651.852652491" Sep 10 16:45:01.144032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:01.143963 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" podStartSLOduration=1.14394255 podStartE2EDuration="1.14394255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:45:01.142995913 +0000 UTC m=+1651.869314844" watchObservedRunningTime="2026-09-10 16:45:01.14394255 +0000 UTC m=+1651.870261481" Sep 10 16:45:02.115694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:02.115653 2562 generic.go:356] "Generic (PLEG): container finished" podID="12a186ee-9a9f-41ab-b903-b5d3c31b132b" containerID="a35069963dc2bfd3e2a5fc15c6f7aae46f77860e21f8bb04a4abfb80cb1bbc87" exitCode=0 Sep 10 16:45:02.116125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:02.115738 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" event={"ID":"12a186ee-9a9f-41ab-b903-b5d3c31b132b","Type":"ContainerDied","Data":"a35069963dc2bfd3e2a5fc15c6f7aae46f77860e21f8bb04a4abfb80cb1bbc87"} Sep 10 16:45:02.117295 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:02.117270 2562 generic.go:356] "Generic (PLEG): container finished" podID="2147dd83-38bf-4d08-993b-1419c302a873" containerID="50a3999b35f515553a48457d90ddff2097ed3ee6e23d41a8ee32580be56e0401" exitCode=0 Sep 10 16:45:02.117417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:02.117353 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" event={"ID":"2147dd83-38bf-4d08-993b-1419c302a873","Type":"ContainerDied","Data":"50a3999b35f515553a48457d90ddff2097ed3ee6e23d41a8ee32580be56e0401"} Sep 10 16:45:03.270079 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:03.270049 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" Sep 10 16:45:03.273064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:03.273030 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" Sep 10 16:45:03.494729 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:03.494639 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:04.129555 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:04.129454 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" Sep 10 16:45:04.129555 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:04.129466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs" event={"ID":"12a186ee-9a9f-41ab-b903-b5d3c31b132b","Type":"ContainerDied","Data":"d37fbdd8fa7b14e4910caabcd54047a3b7e1de27037a7c3c43049038190eea4c"} Sep 10 16:45:04.129555 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:04.129519 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37fbdd8fa7b14e4910caabcd54047a3b7e1de27037a7c3c43049038190eea4c" Sep 10 16:45:04.130904 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:04.130881 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" event={"ID":"2147dd83-38bf-4d08-993b-1419c302a873","Type":"ContainerDied","Data":"549c31f3b158f119d24c34d34edccf2a7d0e55ba745e1c84c2f1b5240703d1d1"} Sep 10 16:45:04.131014 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:04.130905 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh" Sep 10 16:45:04.131014 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:04.130920 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="549c31f3b158f119d24c34d34edccf2a7d0e55ba745e1c84c2f1b5240703d1d1" Sep 10 16:45:21.086131 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:21.086095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:23.567225 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:23.567189 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:25.603611 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:25.603567 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:31.713557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:31.713523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:31.815370 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:31.815337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:31.997638 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:31.997605 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:32.332815 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.332717 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc"] Sep 10 16:45:32.333263 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.333244 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a186ee-9a9f-41ab-b903-b5d3c31b132b" containerName="cleanup" Sep 10 16:45:32.333337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.333265 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a186ee-9a9f-41ab-b903-b5d3c31b132b" containerName="cleanup" Sep 10 16:45:32.333337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.333288 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2147dd83-38bf-4d08-993b-1419c302a873" containerName="cleanup" Sep 10 16:45:32.333337 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.333294 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2147dd83-38bf-4d08-993b-1419c302a873" containerName="cleanup" Sep 10 16:45:32.333453 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.333356 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2147dd83-38bf-4d08-993b-1419c302a873" containerName="cleanup" Sep 10 16:45:32.333453 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.333370 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a186ee-9a9f-41ab-b903-b5d3c31b132b" containerName="cleanup" Sep 10 16:45:32.336455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.336438 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" Sep 10 16:45:32.339578 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.339561 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:45:32.339672 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.339655 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:45:32.345502 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.345477 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc"] Sep 10 16:45:32.421192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.421149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f11eff-4619-44cf-93f8-b1a138ee7a1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc\" (UID: \"e8f11eff-4619-44cf-93f8-b1a138ee7a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" Sep 10 16:45:32.522715 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.522676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f11eff-4619-44cf-93f8-b1a138ee7a1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc\" (UID: \"e8f11eff-4619-44cf-93f8-b1a138ee7a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" Sep 10 16:45:32.523352 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.523327 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f11eff-4619-44cf-93f8-b1a138ee7a1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc\" (UID: \"e8f11eff-4619-44cf-93f8-b1a138ee7a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" Sep 10 16:45:32.596781 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.596658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:32.648416 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.648383 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" Sep 10 16:45:32.933155 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:32.933130 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc"] Sep 10 16:45:32.935672 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:45:32.935642 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f11eff_4619_44cf_93f8_b1a138ee7a1d.slice/crio-8b6306f044e51028651f71787939d4642083fe3595d7467427457f5e11d5a4bd WatchSource:0}: Error finding container 8b6306f044e51028651f71787939d4642083fe3595d7467427457f5e11d5a4bd: Status 404 returned error can't find the container with id 8b6306f044e51028651f71787939d4642083fe3595d7467427457f5e11d5a4bd Sep 10 16:45:33.293152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:33.293110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" event={"ID":"e8f11eff-4619-44cf-93f8-b1a138ee7a1d","Type":"ContainerStarted","Data":"76a176ae65faeed4d128f8b557621d4edc824af6558f1f15e077cf851b0e33a5"} Sep 10 16:45:33.293354 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:33.293169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" event={"ID":"e8f11eff-4619-44cf-93f8-b1a138ee7a1d","Type":"ContainerStarted","Data":"8b6306f044e51028651f71787939d4642083fe3595d7467427457f5e11d5a4bd"} Sep 10 16:45:33.316064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:33.315997 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" podStartSLOduration=1.315977553 podStartE2EDuration="1.315977553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:45:33.313171248 +0000 UTC m=+1684.039490177" watchObservedRunningTime="2026-09-10 16:45:33.315977553 +0000 UTC m=+1684.042296483" Sep 10 16:45:33.653726 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:33.653697 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:34.299606 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:34.299568 2562 generic.go:356] "Generic (PLEG): container finished" podID="e8f11eff-4619-44cf-93f8-b1a138ee7a1d" containerID="76a176ae65faeed4d128f8b557621d4edc824af6558f1f15e077cf851b0e33a5" exitCode=0 Sep 10 16:45:34.300064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:34.299655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" event={"ID":"e8f11eff-4619-44cf-93f8-b1a138ee7a1d","Type":"ContainerDied","Data":"76a176ae65faeed4d128f8b557621d4edc824af6558f1f15e077cf851b0e33a5"} Sep 10 16:45:34.480083 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:34.480015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:35.437783 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:35.437759 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" Sep 10 16:45:35.494173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:35.494142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:35.551492 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:35.551457 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f11eff-4619-44cf-93f8-b1a138ee7a1d-maas-api-service-ca\") pod \"e8f11eff-4619-44cf-93f8-b1a138ee7a1d\" (UID: \"e8f11eff-4619-44cf-93f8-b1a138ee7a1d\") " Sep 10 16:45:35.551826 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:35.551802 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f11eff-4619-44cf-93f8-b1a138ee7a1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8f11eff-4619-44cf-93f8-b1a138ee7a1d" (UID: "e8f11eff-4619-44cf-93f8-b1a138ee7a1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:45:35.652622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:35.652533 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f11eff-4619-44cf-93f8-b1a138ee7a1d-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:45:36.183427 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:36.183393 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:36.310759 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:36.310727 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" Sep 10 16:45:36.310933 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:36.310765 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc" event={"ID":"e8f11eff-4619-44cf-93f8-b1a138ee7a1d","Type":"ContainerDied","Data":"8b6306f044e51028651f71787939d4642083fe3595d7467427457f5e11d5a4bd"} Sep 10 16:45:36.310933 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:36.310815 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b6306f044e51028651f71787939d4642083fe3595d7467427457f5e11d5a4bd" Sep 10 16:45:42.818033 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:42.817996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:42.847771 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:42.847739 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz"] Sep 10 16:45:42.847999 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:42.847963 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" podUID="3e276561-19a7-43a2-a262-8403e76ca550" containerName="payload-processing" containerID="cri-o://91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8" gracePeriod=30 Sep 10 16:45:42.919879 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:42.919841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:43.031863 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.031825 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:43.087863 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.087759 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc"] Sep 10 16:45:43.091303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.091274 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86d230-7qfkc"] Sep 10 16:45:43.115677 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.115651 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:45:43.146056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.146014 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:43.233305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.233268 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e276561-19a7-43a2-a262-8403e76ca550-plugins-config-volume\") pod \"3e276561-19a7-43a2-a262-8403e76ca550\" (UID: \"3e276561-19a7-43a2-a262-8403e76ca550\") " Sep 10 16:45:43.233491 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.233413 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm78h\" (UniqueName: \"kubernetes.io/projected/3e276561-19a7-43a2-a262-8403e76ca550-kube-api-access-dm78h\") pod \"3e276561-19a7-43a2-a262-8403e76ca550\" (UID: \"3e276561-19a7-43a2-a262-8403e76ca550\") " Sep 10 16:45:43.233673 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.233647 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e276561-19a7-43a2-a262-8403e76ca550-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e276561-19a7-43a2-a262-8403e76ca550" (UID: "3e276561-19a7-43a2-a262-8403e76ca550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:45:43.235534 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.235503 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e276561-19a7-43a2-a262-8403e76ca550-kube-api-access-dm78h" (OuterVolumeSpecName: "kube-api-access-dm78h") pod "3e276561-19a7-43a2-a262-8403e76ca550" (UID: "3e276561-19a7-43a2-a262-8403e76ca550"). InnerVolumeSpecName "kube-api-access-dm78h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:45:43.334667 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.334624 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm78h\" (UniqueName: \"kubernetes.io/projected/3e276561-19a7-43a2-a262-8403e76ca550-kube-api-access-dm78h\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:45:43.334667 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.334661 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e276561-19a7-43a2-a262-8403e76ca550-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:45:43.349416 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.349332 2562 generic.go:356] "Generic (PLEG): container finished" podID="3e276561-19a7-43a2-a262-8403e76ca550" containerID="91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8" exitCode=0 Sep 10 16:45:43.349416 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.349377 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" Sep 10 16:45:43.349574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.349408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" event={"ID":"3e276561-19a7-43a2-a262-8403e76ca550","Type":"ContainerDied","Data":"91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8"} Sep 10 16:45:43.349574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.349466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz" event={"ID":"3e276561-19a7-43a2-a262-8403e76ca550","Type":"ContainerDied","Data":"4bd2aa5d2d4632908ce36c0f915d686c8306f68d11a8c274b41c29f883e320e7"} Sep 10 16:45:43.349574 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.349488 2562 scope.go:117] "RemoveContainer" containerID="91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8" Sep 10 16:45:43.358702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.358684 2562 scope.go:117] "RemoveContainer" containerID="91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8" Sep 10 16:45:43.358988 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:45:43.358969 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8\": container with ID starting with 91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8 not found: ID does not exist" containerID="91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8" Sep 10 16:45:43.359056 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.358997 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8"} err="failed to get container status \"91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8\": rpc error: code = NotFound desc = could not find container \"91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8\": container with ID starting with 91ee8bdace49ef93fb90332abce4ecba52fc43e3caa9b58051cbb7e563fcc3a8 not found: ID does not exist" Sep 10 16:45:43.377603 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.377575 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz"] Sep 10 16:45:43.384305 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.384280 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86d230-89b7f94df-xtqzz"] Sep 10 16:45:43.456366 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.456328 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:43.528126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.528089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:43.757213 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.757177 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e276561-19a7-43a2-a262-8403e76ca550" path="/var/lib/kubelet/pods/3e276561-19a7-43a2-a262-8403e76ca550/volumes" Sep 10 16:45:43.757547 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:43.757534 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f11eff-4619-44cf-93f8-b1a138ee7a1d" path="/var/lib/kubelet/pods/e8f11eff-4619-44cf-93f8-b1a138ee7a1d/volumes" Sep 10 16:45:44.495902 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:44.495867 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:45.188861 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:45.188826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:46.145380 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:46.145343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:45:49.991359 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:49.989110 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:50.068450 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.068415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:50.439702 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.439655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:50.707599 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.707505 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq"] Sep 10 16:45:50.708005 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.707986 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e276561-19a7-43a2-a262-8403e76ca550" containerName="payload-processing" Sep 10 16:45:50.708138 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.708007 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e276561-19a7-43a2-a262-8403e76ca550" containerName="payload-processing" Sep 10 16:45:50.708138 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.708060 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f11eff-4619-44cf-93f8-b1a138ee7a1d" containerName="revoke-keys" Sep 10 16:45:50.708138 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.708071 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f11eff-4619-44cf-93f8-b1a138ee7a1d" containerName="revoke-keys" Sep 10 16:45:50.708303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.708173 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f11eff-4619-44cf-93f8-b1a138ee7a1d" containerName="revoke-keys" Sep 10 16:45:50.708303 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.708190 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e276561-19a7-43a2-a262-8403e76ca550" containerName="payload-processing" Sep 10 16:45:50.712621 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.712598 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" Sep 10 16:45:50.715695 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.715665 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:45:50.716164 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.716146 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:45:50.723542 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.723518 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq"] Sep 10 16:45:50.807258 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.807223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870778c7-1ab5-4c59-9c0a-e8b784e574bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq\" (UID: \"870778c7-1ab5-4c59-9c0a-e8b784e574bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" Sep 10 16:45:50.907933 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.907894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870778c7-1ab5-4c59-9c0a-e8b784e574bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq\" (UID: \"870778c7-1ab5-4c59-9c0a-e8b784e574bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" Sep 10 16:45:50.908547 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:50.908527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870778c7-1ab5-4c59-9c0a-e8b784e574bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq\" (UID: \"870778c7-1ab5-4c59-9c0a-e8b784e574bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" Sep 10 16:45:51.023537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:51.023489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" Sep 10 16:45:51.297373 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:51.297335 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq"] Sep 10 16:45:51.298722 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:45:51.298700 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870778c7_1ab5_4c59_9c0a_e8b784e574bf.slice/crio-33a002efd33777885e7f4fe082df37bee3402ead86039094dc30695349819ced WatchSource:0}: Error finding container 33a002efd33777885e7f4fe082df37bee3402ead86039094dc30695349819ced: Status 404 returned error can't find the container with id 33a002efd33777885e7f4fe082df37bee3402ead86039094dc30695349819ced Sep 10 16:45:51.394811 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:51.394775 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" event={"ID":"870778c7-1ab5-4c59-9c0a-e8b784e574bf","Type":"ContainerStarted","Data":"33a002efd33777885e7f4fe082df37bee3402ead86039094dc30695349819ced"} Sep 10 16:45:51.591570 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:51.591536 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:52.381349 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:52.381315 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:52.401335 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:52.401301 2562 generic.go:356] "Generic (PLEG): container finished" podID="870778c7-1ab5-4c59-9c0a-e8b784e574bf" containerID="71cb90f7769149cdf71808062ec7b87b7003635ef6259ca3ba05071783b6bfa4" exitCode=0 Sep 10 16:45:52.401500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:52.401347 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" event={"ID":"870778c7-1ab5-4c59-9c0a-e8b784e574bf","Type":"ContainerDied","Data":"71cb90f7769149cdf71808062ec7b87b7003635ef6259ca3ba05071783b6bfa4"} Sep 10 16:45:53.292414 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:53.292383 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:45:53.540203 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:53.540179 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" Sep 10 16:45:53.631965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:53.631872 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870778c7-1ab5-4c59-9c0a-e8b784e574bf-maas-api-service-ca\") pod \"870778c7-1ab5-4c59-9c0a-e8b784e574bf\" (UID: \"870778c7-1ab5-4c59-9c0a-e8b784e574bf\") " Sep 10 16:45:53.632258 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:53.632235 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870778c7-1ab5-4c59-9c0a-e8b784e574bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "870778c7-1ab5-4c59-9c0a-e8b784e574bf" (UID: "870778c7-1ab5-4c59-9c0a-e8b784e574bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:45:53.733173 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:53.733123 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/870778c7-1ab5-4c59-9c0a-e8b784e574bf-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:45:54.413423 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:54.413393 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" Sep 10 16:45:54.413598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:54.413392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq" event={"ID":"870778c7-1ab5-4c59-9c0a-e8b784e574bf","Type":"ContainerDied","Data":"33a002efd33777885e7f4fe082df37bee3402ead86039094dc30695349819ced"} Sep 10 16:45:54.413598 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:45:54.413539 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a002efd33777885e7f4fe082df37bee3402ead86039094dc30695349819ced" Sep 10 16:46:01.246649 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.246606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:01.251493 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.251463 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs"] Sep 10 16:46:01.256742 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.256717 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1925d8-29817645-7g6qs"] Sep 10 16:46:01.285599 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.285565 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6"] Sep 10 16:46:01.285874 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.285825 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" podUID="1bcda90f-ddd8-4aae-808d-12fe98ac6cc1" containerName="payload-processing" containerID="cri-o://9a899e10d6de62d0017dd765f0b7b70c6ccada04312bf9fca87d50352cee4204" gracePeriod=30 Sep 10 16:46:01.357028 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.356990 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:01.451837 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.451805 2562 generic.go:356] "Generic (PLEG): container finished" podID="1bcda90f-ddd8-4aae-808d-12fe98ac6cc1" containerID="9a899e10d6de62d0017dd765f0b7b70c6ccada04312bf9fca87d50352cee4204" exitCode=0 Sep 10 16:46:01.452002 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.451881 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" event={"ID":"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1","Type":"ContainerDied","Data":"9a899e10d6de62d0017dd765f0b7b70c6ccada04312bf9fca87d50352cee4204"} Sep 10 16:46:01.456341 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.456312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:01.481489 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.481429 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq"] Sep 10 16:46:01.484625 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.484598 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1925d8-2hgjq"] Sep 10 16:46:01.539869 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.539843 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:46:01.559522 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.559487 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:01.611862 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.611826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j46z7\" (UniqueName: \"kubernetes.io/projected/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-kube-api-access-j46z7\") pod \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\" (UID: \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\") " Sep 10 16:46:01.612078 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.611907 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-plugins-config-volume\") pod \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\" (UID: \"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1\") " Sep 10 16:46:01.612717 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.612644 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1bcda90f-ddd8-4aae-808d-12fe98ac6cc1" (UID: "1bcda90f-ddd8-4aae-808d-12fe98ac6cc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:46:01.615537 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.615501 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-kube-api-access-j46z7" (OuterVolumeSpecName: "kube-api-access-j46z7") pod "1bcda90f-ddd8-4aae-808d-12fe98ac6cc1" (UID: "1bcda90f-ddd8-4aae-808d-12fe98ac6cc1"). InnerVolumeSpecName "kube-api-access-j46z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:46:01.713365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.713306 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-plugins-config-volume\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:01.713365 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.713358 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46z7\" (UniqueName: \"kubernetes.io/projected/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1-kube-api-access-j46z7\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:01.757864 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.757828 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a186ee-9a9f-41ab-b903-b5d3c31b132b" path="/var/lib/kubelet/pods/12a186ee-9a9f-41ab-b903-b5d3c31b132b/volumes" Sep 10 16:46:01.758166 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.758153 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="870778c7-1ab5-4c59-9c0a-e8b784e574bf" path="/var/lib/kubelet/pods/870778c7-1ab5-4c59-9c0a-e8b784e574bf/volumes" Sep 10 16:46:01.870021 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.869907 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:01.995765 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:01.995732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:02.458578 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:02.458545 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" Sep 10 16:46:02.459059 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:02.458587 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6" event={"ID":"1bcda90f-ddd8-4aae-808d-12fe98ac6cc1","Type":"ContainerDied","Data":"cc9ef44cf0d9ff8d0e1cc76b66acdebe0654ea1de34ac34b03d68f92459e72c8"} Sep 10 16:46:02.459059 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:02.458650 2562 scope.go:117] "RemoveContainer" containerID="9a899e10d6de62d0017dd765f0b7b70c6ccada04312bf9fca87d50352cee4204" Sep 10 16:46:02.478475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:02.478438 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6"] Sep 10 16:46:02.485586 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:02.485557 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1925d8-544cbdb8c7-7z6c6"] Sep 10 16:46:02.791070 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:02.791004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:03.757512 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:03.757481 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bcda90f-ddd8-4aae-808d-12fe98ac6cc1" path="/var/lib/kubelet/pods/1bcda90f-ddd8-4aae-808d-12fe98ac6cc1/volumes" Sep 10 16:46:04.369790 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:04.369754 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:46:08.574644 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:08.574608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:08.693746 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:08.693711 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:08.937271 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:08.937128 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:09.190460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.190379 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j"] Sep 10 16:46:09.190780 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.190767 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="870778c7-1ab5-4c59-9c0a-e8b784e574bf" containerName="revoke-keys" Sep 10 16:46:09.190820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.190781 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="870778c7-1ab5-4c59-9c0a-e8b784e574bf" containerName="revoke-keys" Sep 10 16:46:09.190820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.190807 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bcda90f-ddd8-4aae-808d-12fe98ac6cc1" containerName="payload-processing" Sep 10 16:46:09.190820 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.190813 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bcda90f-ddd8-4aae-808d-12fe98ac6cc1" containerName="payload-processing" Sep 10 16:46:09.190910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.190865 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="870778c7-1ab5-4c59-9c0a-e8b784e574bf" containerName="revoke-keys" Sep 10 16:46:09.190910 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.190878 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bcda90f-ddd8-4aae-808d-12fe98ac6cc1" containerName="payload-processing" Sep 10 16:46:09.195119 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.195099 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" Sep 10 16:46:09.199444 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.199426 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:46:09.199582 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.199564 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plz5v\"" Sep 10 16:46:09.207120 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.207093 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j"] Sep 10 16:46:09.289502 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.289471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a258c5c-831c-4ec0-9dd5-19d45071393b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j\" (UID: \"7a258c5c-831c-4ec0-9dd5-19d45071393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" Sep 10 16:46:09.318511 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.318467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:09.391015 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.390980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a258c5c-831c-4ec0-9dd5-19d45071393b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j\" (UID: \"7a258c5c-831c-4ec0-9dd5-19d45071393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" Sep 10 16:46:09.391593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.391573 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a258c5c-831c-4ec0-9dd5-19d45071393b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j\" (UID: \"7a258c5c-831c-4ec0-9dd5-19d45071393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" Sep 10 16:46:09.505562 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.505529 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" Sep 10 16:46:09.781813 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:09.781727 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j"] Sep 10 16:46:09.785133 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:46:09.785099 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a258c5c_831c_4ec0_9dd5_19d45071393b.slice/crio-bd0ffb9247c27cf63f842b5b47194510e982b1b9727fdcf8fb75002c2941c81e WatchSource:0}: Error finding container bd0ffb9247c27cf63f842b5b47194510e982b1b9727fdcf8fb75002c2941c81e: Status 404 returned error can't find the container with id bd0ffb9247c27cf63f842b5b47194510e982b1b9727fdcf8fb75002c2941c81e Sep 10 16:46:10.136029 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:10.135948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:10.503967 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:10.503934 2562 generic.go:356] "Generic (PLEG): container finished" podID="7a258c5c-831c-4ec0-9dd5-19d45071393b" containerID="1362d62b6e9d77208c44df2e49e44268780fa3efc046dd58f1d657b93025f64b" exitCode=0 Sep 10 16:46:10.504156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:10.504026 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" event={"ID":"7a258c5c-831c-4ec0-9dd5-19d45071393b","Type":"ContainerDied","Data":"1362d62b6e9d77208c44df2e49e44268780fa3efc046dd58f1d657b93025f64b"} Sep 10 16:46:10.504156 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:10.504090 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" event={"ID":"7a258c5c-831c-4ec0-9dd5-19d45071393b","Type":"ContainerStarted","Data":"bd0ffb9247c27cf63f842b5b47194510e982b1b9727fdcf8fb75002c2941c81e"} Sep 10 16:46:11.098855 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:11.098797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:11.640682 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:11.640657 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" Sep 10 16:46:11.716118 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:11.716084 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a258c5c-831c-4ec0-9dd5-19d45071393b-maas-api-service-ca\") pod \"7a258c5c-831c-4ec0-9dd5-19d45071393b\" (UID: \"7a258c5c-831c-4ec0-9dd5-19d45071393b\") " Sep 10 16:46:11.716459 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:11.716437 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a258c5c-831c-4ec0-9dd5-19d45071393b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a258c5c-831c-4ec0-9dd5-19d45071393b" (UID: "7a258c5c-831c-4ec0-9dd5-19d45071393b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:46:11.817697 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:11.817660 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a258c5c-831c-4ec0-9dd5-19d45071393b-maas-api-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:12.516231 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:12.516198 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" Sep 10 16:46:12.516231 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:12.516208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j" event={"ID":"7a258c5c-831c-4ec0-9dd5-19d45071393b","Type":"ContainerDied","Data":"bd0ffb9247c27cf63f842b5b47194510e982b1b9727fdcf8fb75002c2941c81e"} Sep 10 16:46:12.516671 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:12.516264 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd0ffb9247c27cf63f842b5b47194510e982b1b9727fdcf8fb75002c2941c81e" Sep 10 16:46:16.340501 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:16.340465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:16.442314 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:16.442280 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:19.728281 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:19.728232 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh"] Sep 10 16:46:19.731240 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:19.731202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:19.731547 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:19.731514 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3ccbd4-29817645-s7csh"] Sep 10 16:46:19.761685 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:19.761648 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2147dd83-38bf-4d08-993b-1419c302a873" path="/var/lib/kubelet/pods/2147dd83-38bf-4d08-993b-1419c302a873/volumes" Sep 10 16:46:19.829823 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:19.829790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:19.922307 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:19.922268 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j"] Sep 10 16:46:19.924479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:19.924445 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:19.924712 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:19.924691 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3ccbd4-vbb4j"] Sep 10 16:46:20.009221 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:20.009176 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:20.360612 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:20.360530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:20.489839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:20.489803 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:21.756826 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:21.756792 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a258c5c-831c-4ec0-9dd5-19d45071393b" path="/var/lib/kubelet/pods/7a258c5c-831c-4ec0-9dd5-19d45071393b/volumes" Sep 10 16:46:22.688455 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:22.688419 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:46:26.998832 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:26.998793 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:27.133136 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:27.133095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:27.253976 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:27.253884 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc"] Sep 10 16:46:27.254172 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:27.254131 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" containerID="cri-o://24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86" gracePeriod=30 Sep 10 16:46:27.291160 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:27.291120 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:46:31.552773 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:31.552734 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:46:32.513829 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.513803 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:46:32.630212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.630120 2562 generic.go:356] "Generic (PLEG): container finished" podID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerID="24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86" exitCode=0 Sep 10 16:46:32.630212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.630178 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" Sep 10 16:46:32.630212 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.630201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" event={"ID":"9d6b2e4f-73bd-4958-aafa-f8f5967d9659","Type":"ContainerDied","Data":"24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86"} Sep 10 16:46:32.630667 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.630258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc" event={"ID":"9d6b2e4f-73bd-4958-aafa-f8f5967d9659","Type":"ContainerDied","Data":"d1693713f25439165d8e62ebed45fb9b2cb35137fb704ac8e4a9ae11098a7519"} Sep 10 16:46:32.630667 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.630285 2562 scope.go:117] "RemoveContainer" containerID="24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86" Sep 10 16:46:32.632550 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632529 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-envoy\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632568 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-certs\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632602 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-socket\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632628 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-podinfo\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632665 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632659 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-data\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632682 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-token\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632781 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-credential-socket\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632821 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b98tt\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-kube-api-access-b98tt\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632866 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istiod-ca-cert\") pod \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\" (UID: \"9d6b2e4f-73bd-4958-aafa-f8f5967d9659\") " Sep 10 16:46:32.632877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632865 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:46:32.633141 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.632889 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:46:32.633141 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.633061 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:46:32.633314 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.633293 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-credential-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.633401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.633315 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.633401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.633313 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:46:32.633401 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.633329 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-workload-socket\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.633557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.633400 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:46:32.635134 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.635112 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:46:32.635259 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.635237 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:46:32.635311 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.635237 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:46:32.635417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.635397 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-kube-api-access-b98tt" (OuterVolumeSpecName: "kube-api-access-b98tt") pod "9d6b2e4f-73bd-4958-aafa-f8f5967d9659" (UID: "9d6b2e4f-73bd-4958-aafa-f8f5967d9659"). InnerVolumeSpecName "kube-api-access-b98tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:46:32.655777 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.655747 2562 scope.go:117] "RemoveContainer" containerID="7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725" Sep 10 16:46:32.667192 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.667162 2562 scope.go:117] "RemoveContainer" containerID="24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86" Sep 10 16:46:32.667544 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:46:32.667519 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86\": container with ID starting with 24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86 not found: ID does not exist" containerID="24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86" Sep 10 16:46:32.667643 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.667555 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86"} err="failed to get container status \"24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86\": rpc error: code = NotFound desc = could not find container \"24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86\": container with ID starting with 24aa20ff51f9b2cecc9f09ca9f8533c0b5fb8c3dcaf0eac1a4248d1fcf01ed86 not found: ID does not exist" Sep 10 16:46:32.667643 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.667574 2562 scope.go:117] "RemoveContainer" containerID="7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725" Sep 10 16:46:32.667955 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:46:32.667935 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725\": container with ID starting with 7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725 not found: ID does not exist" containerID="7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725" Sep 10 16:46:32.668095 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.667958 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725"} err="failed to get container status \"7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725\": rpc error: code = NotFound desc = could not find container \"7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725\": container with ID starting with 7129bbb3b4572e38d21da38b83a8d0b9c094093bb7f0727321fdf743b7e7e725 not found: ID does not exist" Sep 10 16:46:32.734740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.734701 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-data\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.734740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.734734 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-token\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.734740 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.734746 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98tt\" (UniqueName: \"kubernetes.io/projected/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-kube-api-access-b98tt\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.734987 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.734756 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istiod-ca-cert\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.734987 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.734768 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-envoy\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.734987 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.734779 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6b2e4f-73bd-4958-aafa-f8f5967d9659-istio-podinfo\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:46:32.958241 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.958207 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc"] Sep 10 16:46:32.962308 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:32.962278 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3ccbd4-openshift-default-5944fc784-cd4fc"] Sep 10 16:46:33.757417 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:46:33.757386 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" path="/var/lib/kubelet/pods/9d6b2e4f-73bd-4958-aafa-f8f5967d9659/volumes" Sep 10 16:47:29.856533 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:47:29.856497 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:47:29.859151 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:47:29.859129 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:47:52.945703 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:47:52.945664 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:47:53.387600 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:47:53.387563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:47:54.955223 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:47:54.955187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:48:18.845873 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:48:18.845837 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:48:26.959362 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:48:26.959323 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:48:34.242032 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:48:34.241994 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:48:42.338280 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:48:42.338234 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:48:55.335728 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:48:55.335685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:49:03.348797 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:49:03.348758 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:49:04.069666 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:49:04.069582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:49:57.843949 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:49:57.843917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:49:57.945250 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:49:57.945204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:50:06.003870 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:06.003808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:50:06.105738 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:06.105702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:50:30.217690 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:30.217613 2562 scope.go:117] "RemoveContainer" containerID="f850e03af5cb3928ea63380667b5b6b17215c2ed20e24644cd14540e7a86777b" Sep 10 16:50:31.444557 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:31.444520 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:50:38.350986 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:38.350935 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:50:42.952161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:42.952124 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:50:45.841608 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:45.841577 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:50:57.078355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:57.078318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:50:59.188159 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:50:59.188123 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:51:00.150965 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:51:00.150928 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:51:30.238600 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:51:30.238565 2562 scope.go:117] "RemoveContainer" containerID="50a3999b35f515553a48457d90ddff2097ed3ee6e23d41a8ee32580be56e0401" Sep 10 16:51:30.246816 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:51:30.246791 2562 scope.go:117] "RemoveContainer" containerID="a35069963dc2bfd3e2a5fc15c6f7aae46f77860e21f8bb04a4abfb80cb1bbc87" Sep 10 16:51:44.383321 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:51:44.383285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:51:45.400932 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:51:45.400894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:51:45.547408 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:51:45.547373 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:51:52.436743 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:51:52.436657 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:51:58.548382 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:51:58.548337 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:52:00.560842 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:00.560808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:52:00.621844 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:00.621810 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:52:06.346125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.346087 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:52:06.541002 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.540965 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-85dh9"] Sep 10 16:52:06.541606 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.541589 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a258c5c-831c-4ec0-9dd5-19d45071393b" containerName="revoke-keys" Sep 10 16:52:06.541653 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.541611 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a258c5c-831c-4ec0-9dd5-19d45071393b" containerName="revoke-keys" Sep 10 16:52:06.541653 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.541636 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" Sep 10 16:52:06.541653 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.541645 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" Sep 10 16:52:06.541782 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.541769 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" Sep 10 16:52:06.541818 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.541796 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a258c5c-831c-4ec0-9dd5-19d45071393b" containerName="revoke-keys" Sep 10 16:52:06.545289 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.545267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.548463 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.548438 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 16:52:06.548594 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.548522 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:52:06.549786 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.549770 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5q8xx\"" Sep 10 16:52:06.549923 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.549907 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 16:52:06.550011 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.549975 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 16:52:06.556087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.556063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-85dh9"] Sep 10 16:52:06.605733 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.605639 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-metrics-certs\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.605733 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.605719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-maas-api-tls\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.605921 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.605754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpcf\" (UniqueName: \"kubernetes.io/projected/49b070a1-559b-4f07-82f2-82b68c9a222e-kube-api-access-mtpcf\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.605921 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.605810 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b070a1-559b-4f07-82f2-82b68c9a222e-openshift-service-ca\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.706525 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.706480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-maas-api-tls\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.706742 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.706545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpcf\" (UniqueName: \"kubernetes.io/projected/49b070a1-559b-4f07-82f2-82b68c9a222e-kube-api-access-mtpcf\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.706742 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.706603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b070a1-559b-4f07-82f2-82b68c9a222e-openshift-service-ca\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.706742 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.706669 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-metrics-certs\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.707319 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.707296 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b070a1-559b-4f07-82f2-82b68c9a222e-openshift-service-ca\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.709126 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.709096 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-maas-api-tls\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.709224 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.709095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-metrics-certs\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.715049 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.715012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpcf\" (UniqueName: \"kubernetes.io/projected/49b070a1-559b-4f07-82f2-82b68c9a222e-kube-api-access-mtpcf\") pod \"maas-api-6c68568df5-85dh9\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:06.859429 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:06.859328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:07.134775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:07.134685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-85dh9"] Sep 10 16:52:07.137643 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:52:07.137618 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b070a1_559b_4f07_82f2_82b68c9a222e.slice/crio-cb499f125477f89befbc8e71ece82304ea66fde228eb271053c2c78749b0ee32 WatchSource:0}: Error finding container cb499f125477f89befbc8e71ece82304ea66fde228eb271053c2c78749b0ee32: Status 404 returned error can't find the container with id cb499f125477f89befbc8e71ece82304ea66fde228eb271053c2c78749b0ee32 Sep 10 16:52:07.139066 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:07.139028 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:52:07.370860 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:07.370818 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" event={"ID":"49b070a1-559b-4f07-82f2-82b68c9a222e","Type":"ContainerStarted","Data":"cb499f125477f89befbc8e71ece82304ea66fde228eb271053c2c78749b0ee32"} Sep 10 16:52:10.389568 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:10.389535 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" event={"ID":"49b070a1-559b-4f07-82f2-82b68c9a222e","Type":"ContainerStarted","Data":"a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7"} Sep 10 16:52:10.389969 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:10.389614 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:10.410479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:10.410431 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" podStartSLOduration=2.124903413 podStartE2EDuration="4.410414794s" totalImagesPullingTime="2.285511381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:52:10.409010878 +0000 UTC m=+2081.135329832" watchObservedRunningTime="2026-09-10 16:52:10.410414794 +0000 UTC m=+2081.136733723" Sep 10 16:52:13.825947 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:13.825914 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-85dh9"] Sep 10 16:52:13.826532 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:13.826121 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" podUID="49b070a1-559b-4f07-82f2-82b68c9a222e" containerName="maas-api" containerID="cri-o://a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7" gracePeriod=30 Sep 10 16:52:14.074068 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.074027 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:14.187462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.187361 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b070a1-559b-4f07-82f2-82b68c9a222e-openshift-service-ca\") pod \"49b070a1-559b-4f07-82f2-82b68c9a222e\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " Sep 10 16:52:14.187462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.187424 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-maas-api-tls\") pod \"49b070a1-559b-4f07-82f2-82b68c9a222e\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " Sep 10 16:52:14.187462 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.187455 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-metrics-certs\") pod \"49b070a1-559b-4f07-82f2-82b68c9a222e\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " Sep 10 16:52:14.187718 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.187515 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtpcf\" (UniqueName: \"kubernetes.io/projected/49b070a1-559b-4f07-82f2-82b68c9a222e-kube-api-access-mtpcf\") pod \"49b070a1-559b-4f07-82f2-82b68c9a222e\" (UID: \"49b070a1-559b-4f07-82f2-82b68c9a222e\") " Sep 10 16:52:14.187791 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.187734 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b070a1-559b-4f07-82f2-82b68c9a222e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49b070a1-559b-4f07-82f2-82b68c9a222e" (UID: "49b070a1-559b-4f07-82f2-82b68c9a222e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:52:14.187912 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.187894 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b070a1-559b-4f07-82f2-82b68c9a222e-openshift-service-ca\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:52:14.189892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.189862 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b070a1-559b-4f07-82f2-82b68c9a222e-kube-api-access-mtpcf" (OuterVolumeSpecName: "kube-api-access-mtpcf") pod "49b070a1-559b-4f07-82f2-82b68c9a222e" (UID: "49b070a1-559b-4f07-82f2-82b68c9a222e"). InnerVolumeSpecName "kube-api-access-mtpcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:52:14.189892 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.189860 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49b070a1-559b-4f07-82f2-82b68c9a222e" (UID: "49b070a1-559b-4f07-82f2-82b68c9a222e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:52:14.190070 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.189893 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "49b070a1-559b-4f07-82f2-82b68c9a222e" (UID: "49b070a1-559b-4f07-82f2-82b68c9a222e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:52:14.243208 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.243171 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:52:14.288759 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.288723 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-maas-api-tls\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:52:14.288759 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.288748 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49b070a1-559b-4f07-82f2-82b68c9a222e-metrics-certs\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:52:14.288759 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.288759 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpcf\" (UniqueName: \"kubernetes.io/projected/49b070a1-559b-4f07-82f2-82b68c9a222e-kube-api-access-mtpcf\") on node \"ip-10-0-139-24.ec2.internal\" DevicePath \"\"" Sep 10 16:52:14.411119 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.411078 2562 generic.go:356] "Generic (PLEG): container finished" podID="49b070a1-559b-4f07-82f2-82b68c9a222e" containerID="a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7" exitCode=0 Sep 10 16:52:14.411318 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.411124 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" Sep 10 16:52:14.411318 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.411173 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" event={"ID":"49b070a1-559b-4f07-82f2-82b68c9a222e","Type":"ContainerDied","Data":"a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7"} Sep 10 16:52:14.411318 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.411240 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c68568df5-85dh9" event={"ID":"49b070a1-559b-4f07-82f2-82b68c9a222e","Type":"ContainerDied","Data":"cb499f125477f89befbc8e71ece82304ea66fde228eb271053c2c78749b0ee32"} Sep 10 16:52:14.411318 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.411271 2562 scope.go:117] "RemoveContainer" containerID="a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7" Sep 10 16:52:14.424982 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.424958 2562 scope.go:117] "RemoveContainer" containerID="a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7" Sep 10 16:52:14.425387 ip-10-0-139-24 kubenswrapper[2562]: E0910 16:52:14.425361 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7\": container with ID starting with a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7 not found: ID does not exist" containerID="a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7" Sep 10 16:52:14.425463 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.425398 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7"} err="failed to get container status \"a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7\": rpc error: code = NotFound desc = could not find container \"a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7\": container with ID starting with a04e3c93289a6304f72fbf5cf66666b9e42910e0543c262fa21816235dd115d7 not found: ID does not exist" Sep 10 16:52:14.436475 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.436448 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-85dh9"] Sep 10 16:52:14.440472 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:14.440412 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-85dh9"] Sep 10 16:52:15.757289 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:15.757254 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b070a1-559b-4f07-82f2-82b68c9a222e" path="/var/lib/kubelet/pods/49b070a1-559b-4f07-82f2-82b68c9a222e/volumes" Sep 10 16:52:21.051775 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:21.051741 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:52:27.138980 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:27.138940 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:52:29.891508 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:29.891477 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:52:29.893336 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:29.893313 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:52:30.269727 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:30.269692 2562 scope.go:117] "RemoveContainer" containerID="1362d62b6e9d77208c44df2e49e44268780fa3efc046dd58f1d657b93025f64b" Sep 10 16:52:30.278110 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:30.278083 2562 scope.go:117] "RemoveContainer" containerID="76a176ae65faeed4d128f8b557621d4edc824af6558f1f15e077cf851b0e33a5" Sep 10 16:52:30.286161 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:30.286132 2562 scope.go:117] "RemoveContainer" containerID="71cb90f7769149cdf71808062ec7b87b7003635ef6259ca3ba05071783b6bfa4" Sep 10 16:52:34.343052 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:34.343008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kgq7c"] Sep 10 16:52:42.342793 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:42.342763 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b77968d9-922js_05e1d2a0-ef40-4578-ad49-639e172a3b11/copy-manifests/0.log" Sep 10 16:52:42.355616 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:42.355585 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b77968d9-922js_05e1d2a0-ef40-4578-ad49-639e172a3b11/manager/0.log" Sep 10 16:52:44.545593 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:44.545562 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-kgq7c_f9d37e27-89a2-4009-b1be-9a62058cb899/limitador/0.log" Sep 10 16:52:45.009783 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:45.009750 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz_152ab912-d24c-4e5e-a0a6-ba768c0cb7ec/istio-proxy/0.log" Sep 10 16:52:45.122795 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:45.122760 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p5cjc_31ffffbd-1637-49ae-9e51-f7398e7add88/discovery/0.log" Sep 10 16:52:45.468333 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:45.468242 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v25lq_9ebc2bba-8023-4022-8a30-c5ecae72b843/istio-proxy/1.log" Sep 10 16:52:46.467777 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:46.467748 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-96tt5_16892c0d-d99a-47e9-9164-47144a4de1b5/main/0.log" Sep 10 16:52:46.678789 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:46.678759 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-csjb7_b760bdb8-b675-4ab5-b992-ac6b035a6d28/main/0.log" Sep 10 16:52:52.126712 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:52.126671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:52:52.229460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:52.229414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:52:53.523340 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:53.523309 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" limit=1024 Sep 10 16:52:55.231548 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:55.231514 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l849b_3cb5f5c7-9cca-44c3-9013-6fcc3788f877/global-pull-secret-syncer/0.log" Sep 10 16:52:55.332148 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:55.332121 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4544q_a809f2d7-03d7-487d-a48e-4628fc304d2a/konnectivity-agent/0.log" Sep 10 16:52:55.459773 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:52:55.459744 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-24.ec2.internal_e031a7be8eb499dcf51c16d6ce862b92/haproxy/0.log" Sep 10 16:53:00.391104 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:00.391072 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-kgq7c_f9d37e27-89a2-4009-b1be-9a62058cb899/limitador/0.log" Sep 10 16:53:01.968622 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:01.968590 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42f6a402-417c-46b3-b207-800ecf4e0405/alertmanager/0.log" Sep 10 16:53:01.993543 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:01.993519 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42f6a402-417c-46b3-b207-800ecf4e0405/config-reloader/0.log" Sep 10 16:53:02.018504 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.018478 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42f6a402-417c-46b3-b207-800ecf4e0405/kube-rbac-proxy-web/0.log" Sep 10 16:53:02.044087 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.044025 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42f6a402-417c-46b3-b207-800ecf4e0405/kube-rbac-proxy/0.log" Sep 10 16:53:02.074179 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.074154 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42f6a402-417c-46b3-b207-800ecf4e0405/kube-rbac-proxy-metric/0.log" Sep 10 16:53:02.101566 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.101534 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42f6a402-417c-46b3-b207-800ecf4e0405/prom-label-proxy/0.log" Sep 10 16:53:02.129889 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.129865 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42f6a402-417c-46b3-b207-800ecf4e0405/init-config-reloader/0.log" Sep 10 16:53:02.202931 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.202849 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-d6m2j_bfdeec23-60c7-42ec-9725-940b1e1910ee/kube-state-metrics/0.log" Sep 10 16:53:02.234389 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.234348 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-d6m2j_bfdeec23-60c7-42ec-9725-940b1e1910ee/kube-rbac-proxy-main/0.log" Sep 10 16:53:02.258385 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.258349 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-d6m2j_bfdeec23-60c7-42ec-9725-940b1e1910ee/kube-rbac-proxy-self/0.log" Sep 10 16:53:02.288060 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.288020 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69f846fc4-5gghq_031e2916-b4b3-474e-9041-22ed51454669/metrics-server/0.log" Sep 10 16:53:02.339695 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.339662 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-9c79h_a7b17f6e-bdf2-4093-9e9a-7e76453423ef/monitoring-plugin/0.log" Sep 10 16:53:02.584762 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.584719 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wvwvx_7377ae4f-ad20-483a-98c8-34f6e4d55eca/node-exporter/0.log" Sep 10 16:53:02.651250 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.651221 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wvwvx_7377ae4f-ad20-483a-98c8-34f6e4d55eca/kube-rbac-proxy/0.log" Sep 10 16:53:02.680683 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.680658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wvwvx_7377ae4f-ad20-483a-98c8-34f6e4d55eca/init-textfile/0.log" Sep 10 16:53:02.727577 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.727542 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-4p5cb_fa17ef87-754e-433c-b22d-63e408216cb1/kube-rbac-proxy-main/0.log" Sep 10 16:53:02.751955 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.751927 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-4p5cb_fa17ef87-754e-433c-b22d-63e408216cb1/kube-rbac-proxy-self/0.log" Sep 10 16:53:02.780549 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.780462 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-4p5cb_fa17ef87-754e-433c-b22d-63e408216cb1/openshift-state-metrics/0.log" Sep 10 16:53:02.844233 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.844204 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efcd3fac-a22a-459f-8bb1-04f6165757c3/prometheus/0.log" Sep 10 16:53:02.865644 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.865600 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efcd3fac-a22a-459f-8bb1-04f6165757c3/config-reloader/0.log" Sep 10 16:53:02.890338 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.890308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efcd3fac-a22a-459f-8bb1-04f6165757c3/thanos-sidecar/0.log" Sep 10 16:53:02.915246 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.915216 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efcd3fac-a22a-459f-8bb1-04f6165757c3/kube-rbac-proxy-web/0.log" Sep 10 16:53:02.940949 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.940925 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efcd3fac-a22a-459f-8bb1-04f6165757c3/kube-rbac-proxy/0.log" Sep 10 16:53:02.967320 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.967294 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efcd3fac-a22a-459f-8bb1-04f6165757c3/kube-rbac-proxy-thanos/0.log" Sep 10 16:53:02.998877 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:02.998852 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efcd3fac-a22a-459f-8bb1-04f6165757c3/init-config-reloader/0.log" Sep 10 16:53:03.035053 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.034959 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-4hsv4_d197fbe3-3456-40ab-bad7-20390790af34/prometheus-operator/0.log" Sep 10 16:53:03.056578 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.056551 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-4hsv4_d197fbe3-3456-40ab-bad7-20390790af34/kube-rbac-proxy/0.log" Sep 10 16:53:03.127654 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.127626 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7748f44994-6bksq_5f816f97-e930-4fb4-b514-d070fc2b1686/telemeter-client/0.log" Sep 10 16:53:03.152607 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.152575 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7748f44994-6bksq_5f816f97-e930-4fb4-b514-d070fc2b1686/reload/0.log" Sep 10 16:53:03.177152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.177124 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7748f44994-6bksq_5f816f97-e930-4fb4-b514-d070fc2b1686/kube-rbac-proxy/0.log" Sep 10 16:53:03.216363 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.216332 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546cf76c59-9w54t_a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad/thanos-query/0.log" Sep 10 16:53:03.243895 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.243864 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546cf76c59-9w54t_a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad/kube-rbac-proxy-web/0.log" Sep 10 16:53:03.268908 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.268884 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546cf76c59-9w54t_a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad/kube-rbac-proxy/0.log" Sep 10 16:53:03.293956 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.293892 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546cf76c59-9w54t_a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad/prom-label-proxy/0.log" Sep 10 16:53:03.319397 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.319362 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546cf76c59-9w54t_a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad/kube-rbac-proxy-rules/0.log" Sep 10 16:53:03.348964 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:03.348933 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-546cf76c59-9w54t_a05dc8a2-cb9e-4932-b7da-b71c3ee8fcad/kube-rbac-proxy-metrics/0.log" Sep 10 16:53:05.651343 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.651303 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8"] Sep 10 16:53:05.651839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.651778 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b070a1-559b-4f07-82f2-82b68c9a222e" containerName="maas-api" Sep 10 16:53:05.651839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.651799 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b070a1-559b-4f07-82f2-82b68c9a222e" containerName="maas-api" Sep 10 16:53:05.651839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.651820 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" Sep 10 16:53:05.651839 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.651827 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" Sep 10 16:53:05.652125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.651909 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b070a1-559b-4f07-82f2-82b68c9a222e" containerName="maas-api" Sep 10 16:53:05.652125 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.651922 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6b2e4f-73bd-4958-aafa-f8f5967d9659" containerName="istio-proxy" Sep 10 16:53:05.655119 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.655096 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.657927 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.657907 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbsrw\"/\"kube-root-ca.crt\"" Sep 10 16:53:05.659226 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.659207 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbsrw\"/\"openshift-service-ca.crt\"" Sep 10 16:53:05.659226 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.659230 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rbsrw\"/\"default-dockercfg-8nc6g\"" Sep 10 16:53:05.663457 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.663436 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8"] Sep 10 16:53:05.795005 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.794964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-proc\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.795005 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.795005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-podres\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.795298 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.795064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhk9m\" (UniqueName: \"kubernetes.io/projected/1e369cc4-3067-49de-bc4c-50c6023495bc-kube-api-access-fhk9m\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.795298 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.795187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-lib-modules\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.795399 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.795301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-sys\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896474 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhk9m\" (UniqueName: \"kubernetes.io/projected/1e369cc4-3067-49de-bc4c-50c6023495bc-kube-api-access-fhk9m\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-lib-modules\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896547 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-sys\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-proc\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-podres\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896678 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896664 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-sys\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896922 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896672 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-proc\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896922 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896718 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-podres\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.896922 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.896723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e369cc4-3067-49de-bc4c-50c6023495bc-lib-modules\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.907623 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.907541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhk9m\" (UniqueName: \"kubernetes.io/projected/1e369cc4-3067-49de-bc4c-50c6023495bc-kube-api-access-fhk9m\") pod \"perf-node-gather-daemonset-2vrf8\" (UID: \"1e369cc4-3067-49de-bc4c-50c6023495bc\") " pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:05.966781 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:05.966736 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:06.240698 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:06.240665 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8"] Sep 10 16:53:06.243000 ip-10-0-139-24 kubenswrapper[2562]: W0910 16:53:06.242974 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e369cc4_3067_49de_bc4c_50c6023495bc.slice/crio-796cd65ee03a2a7124a8031af7583f59dc79fc570d8882253293b6a2dca54dc7 WatchSource:0}: Error finding container 796cd65ee03a2a7124a8031af7583f59dc79fc570d8882253293b6a2dca54dc7: Status 404 returned error can't find the container with id 796cd65ee03a2a7124a8031af7583f59dc79fc570d8882253293b6a2dca54dc7 Sep 10 16:53:06.673453 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:06.673416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" event={"ID":"1e369cc4-3067-49de-bc4c-50c6023495bc","Type":"ContainerStarted","Data":"0f1c50515440e927cedae215b82193ee15b5b95752dcbe7f01706f295d5b6d76"} Sep 10 16:53:06.673859 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:06.673468 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" event={"ID":"1e369cc4-3067-49de-bc4c-50c6023495bc","Type":"ContainerStarted","Data":"796cd65ee03a2a7124a8031af7583f59dc79fc570d8882253293b6a2dca54dc7"} Sep 10 16:53:06.673859 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:06.673546 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:06.696528 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:06.695314 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" podStartSLOduration=1.6952818 podStartE2EDuration="1.6952818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:53:06.692448696 +0000 UTC m=+2137.418767624" watchObservedRunningTime="2026-09-10 16:53:06.6952818 +0000 UTC m=+2137.421600743" Sep 10 16:53:07.671798 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:07.671763 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sb6w5_e2311967-fbe1-4acc-8e9b-1145db261669/dns/0.log" Sep 10 16:53:07.734355 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:07.734324 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sb6w5_e2311967-fbe1-4acc-8e9b-1145db261669/kube-rbac-proxy/0.log" Sep 10 16:53:07.811348 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:07.811317 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jv76_dff7f8c0-5a53-45bc-8b61-8abebb4bf30c/dns-node-resolver/0.log" Sep 10 16:53:08.451479 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:08.451448 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcxpt_0f12a9ba-2874-4c63-803e-b40f109453ac/node-ca/0.log" Sep 10 16:53:09.421403 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:09.421371 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz_152ab912-d24c-4e5e-a0a6-ba768c0cb7ec/istio-proxy/0.log" Sep 10 16:53:09.726182 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:09.726102 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p5cjc_31ffffbd-1637-49ae-9e51-f7398e7add88/discovery/0.log" Sep 10 16:53:09.803610 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:09.803573 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v25lq_9ebc2bba-8023-4022-8a30-c5ecae72b843/istio-proxy/0.log" Sep 10 16:53:09.811749 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:09.811723 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v25lq_9ebc2bba-8023-4022-8a30-c5ecae72b843/istio-proxy/1.log" Sep 10 16:53:10.849226 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:10.849194 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tmhw5_f3531b68-1218-45f2-8e39-c7b7b948ecc7/serve-healthcheck-canary/0.log" Sep 10 16:53:11.347080 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:11.347031 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmfkp_8479af56-72f9-4e42-88ff-2f87408a9455/kube-rbac-proxy/0.log" Sep 10 16:53:11.371470 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:11.371445 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmfkp_8479af56-72f9-4e42-88ff-2f87408a9455/exporter/0.log" Sep 10 16:53:11.394364 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:11.394334 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmfkp_8479af56-72f9-4e42-88ff-2f87408a9455/extractor/0.log" Sep 10 16:53:12.688395 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:12.688366 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rbsrw/perf-node-gather-daemonset-2vrf8" Sep 10 16:53:14.230996 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:14.230939 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b77968d9-922js_05e1d2a0-ef40-4578-ad49-639e172a3b11/manager/0.log" Sep 10 16:53:14.252970 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:14.252945 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b77968d9-922js_05e1d2a0-ef40-4578-ad49-639e172a3b11/copy-manifests/0.log" Sep 10 16:53:15.605853 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:15.605766 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-575575bffc-q55pv_ccc2300e-db24-4ad7-9d65-99eba08bbcc4/manager/0.log" Sep 10 16:53:15.663447 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:15.663417 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kchdg_decffee1-2d2a-4fc9-9c45-9c7aaa184066/openshift-lws-operator/0.log" Sep 10 16:53:15.901218 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:15.901190 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz_152ab912-d24c-4e5e-a0a6-ba768c0cb7ec/istio-proxy/0.log" Sep 10 16:53:16.226158 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:16.226130 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p5cjc_31ffffbd-1637-49ae-9e51-f7398e7add88/discovery/0.log" Sep 10 16:53:16.280890 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:16.280857 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v25lq_9ebc2bba-8023-4022-8a30-c5ecae72b843/istio-proxy/0.log" Sep 10 16:53:16.289064 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:16.289022 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v25lq_9ebc2bba-8023-4022-8a30-c5ecae72b843/istio-proxy/1.log" Sep 10 16:53:17.942516 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:17.942488 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7f8bz_152ab912-d24c-4e5e-a0a6-ba768c0cb7ec/istio-proxy/0.log" Sep 10 16:53:18.248833 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:18.248803 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p5cjc_31ffffbd-1637-49ae-9e51-f7398e7add88/discovery/0.log" Sep 10 16:53:18.304738 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:18.304658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v25lq_9ebc2bba-8023-4022-8a30-c5ecae72b843/istio-proxy/0.log" Sep 10 16:53:18.312460 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:18.312434 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v25lq_9ebc2bba-8023-4022-8a30-c5ecae72b843/istio-proxy/1.log" Sep 10 16:53:22.687835 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:22.687806 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jvhqn_ad37714e-ec26-404b-8b83-76e78c1d12a5/kube-multus-additional-cni-plugins/0.log" Sep 10 16:53:22.712152 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:22.712119 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jvhqn_ad37714e-ec26-404b-8b83-76e78c1d12a5/egress-router-binary-copy/0.log" Sep 10 16:53:22.735694 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:22.735658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jvhqn_ad37714e-ec26-404b-8b83-76e78c1d12a5/cni-plugins/0.log" Sep 10 16:53:22.757304 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:22.757278 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jvhqn_ad37714e-ec26-404b-8b83-76e78c1d12a5/bond-cni-plugin/0.log" Sep 10 16:53:22.777999 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:22.777973 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jvhqn_ad37714e-ec26-404b-8b83-76e78c1d12a5/routeoverride-cni/0.log" Sep 10 16:53:22.799265 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:22.799233 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jvhqn_ad37714e-ec26-404b-8b83-76e78c1d12a5/whereabouts-cni-bincopy/0.log" Sep 10 16:53:22.819843 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:22.819814 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jvhqn_ad37714e-ec26-404b-8b83-76e78c1d12a5/whereabouts-cni/0.log" Sep 10 16:53:23.012345 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:23.012311 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ppcww_8c743517-72d7-4bb4-bda1-eb8c9fbd2bbd/kube-multus/0.log" Sep 10 16:53:23.077661 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:23.077636 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hn92b_944b5648-71ef-4980-927f-351334db589e/network-metrics-daemon/0.log" Sep 10 16:53:23.097013 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:23.096984 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hn92b_944b5648-71ef-4980-927f-351334db589e/kube-rbac-proxy/0.log" Sep 10 16:53:23.960531 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:23.960503 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-controller/0.log" Sep 10 16:53:23.977699 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:23.977668 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/0.log" Sep 10 16:53:23.977851 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:23.977705 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovn-acl-logging/1.log" Sep 10 16:53:23.995794 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:23.995768 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/kube-rbac-proxy-node/0.log" Sep 10 16:53:24.018374 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:24.018351 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 16:53:24.042368 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:24.042339 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/northd/0.log" Sep 10 16:53:24.064500 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:24.064471 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/nbdb/0.log" Sep 10 16:53:24.086946 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:24.086920 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/sbdb/0.log" Sep 10 16:53:24.140028 ip-10-0-139-24 kubenswrapper[2562]: I0910 16:53:24.139995 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h2jpq_59c257de-f095-4753-adb3-14789e3b440d/ovnkube-controller/0.log"