Sep 11 19:27:18.926699 ip-10-0-138-82 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 19:27:18.926832 ip-10-0-138-82 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 19:27:18.926847 ip-10-0-138-82 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 19:27:18.927060 ip-10-0-138-82 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 19:27:29.087232 ip-10-0-138-82 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 19:27:29.087250 ip-10-0-138-82 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 63be930337934fe7928d3eaab7dcc205 -- Sep 11 19:29:25.373292 ip-10-0-138-82 systemd[1]: Starting Kubernetes Kubelet... Sep 11 19:29:25.799759 ip-10-0-138-82 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 19:29:25.799759 ip-10-0-138-82 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 19:29:25.799759 ip-10-0-138-82 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 19:29:25.799759 ip-10-0-138-82 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 19:29:25.799759 ip-10-0-138-82 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 19:29:25.801297 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.801210 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 19:29:25.805581 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805562 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:29:25.805581 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805581 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805585 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805589 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805592 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805595 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805597 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805600 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805603 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805605 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805609 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805612 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805614 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805617 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805619 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805622 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805625 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805627 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805630 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805632 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805636 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:29:25.805645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805638 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805641 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805644 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805647 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805650 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805653 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805656 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805659 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805662 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805664 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805667 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805669 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805672 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805674 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805677 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805680 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805682 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805684 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805687 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:29:25.806134 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805689 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805692 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805694 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805697 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805700 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805702 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805705 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805707 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805710 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805712 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805715 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805717 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805720 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805723 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805726 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805729 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805731 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805734 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805736 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805739 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:29:25.806596 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805741 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805744 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805746 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805749 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805751 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805754 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805756 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805759 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805761 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805765 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805767 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805770 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805772 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805775 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805777 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805780 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805782 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805785 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805788 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805790 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:29:25.807081 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805793 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805795 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805798 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805800 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805802 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805807 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805811 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805816 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805822 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805825 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805828 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805832 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805836 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805839 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805841 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805844 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805846 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805848 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.805851 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:29:25.807887 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806603 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806612 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806615 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806618 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806622 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806624 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806627 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806630 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806633 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806636 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806639 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806642 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806644 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806647 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806649 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806652 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806655 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806657 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806661 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:29:25.808384 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806665 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806668 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806670 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806673 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806676 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806678 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806681 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806684 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806686 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806689 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806691 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806694 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806696 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806698 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806701 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806704 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806706 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806709 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806712 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806715 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:29:25.808857 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806718 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806720 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806723 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806725 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806728 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806730 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806734 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806737 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806740 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806743 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806746 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806749 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806753 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806756 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806759 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806762 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806764 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806767 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806769 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:29:25.809364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806772 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806774 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806777 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806779 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806783 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806787 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806789 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806792 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806795 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806797 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806800 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806802 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806805 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806808 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806810 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806813 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806815 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806819 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806822 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806824 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:29:25.809834 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806826 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806829 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806831 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806835 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806839 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806842 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806844 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806847 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806850 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806852 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806855 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806857 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806860 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806863 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806865 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806868 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806870 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806873 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806875 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806878 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:29:25.810433 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.806880 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808608 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808619 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808622 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808626 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808630 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808634 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808638 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808641 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808643 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808647 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808651 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808654 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808657 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808660 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808663 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808665 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808668 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808671 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:29:25.810925 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808674 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808676 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808679 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808682 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808684 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808687 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808689 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808691 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808697 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808700 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808702 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808705 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808707 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808710 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808713 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808716 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808719 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808721 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808723 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808726 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:29:25.811409 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808729 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808732 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808734 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808737 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808739 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808742 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808744 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808747 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808749 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808752 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808754 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808756 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808759 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808762 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808764 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808767 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808769 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808772 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808774 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:29:25.811899 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808777 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808779 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808782 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808785 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808787 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808790 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808792 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808795 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808799 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808801 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808804 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808806 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808809 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808811 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808815 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808818 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808821 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808824 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808827 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:29:25.812388 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808830 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808833 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808836 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808838 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808841 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808843 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808846 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808848 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808851 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808853 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808856 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808858 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808861 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808864 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808866 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808869 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808872 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808875 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808877 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:29:25.812883 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808880 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808882 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808886 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.808889 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.808978 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809016 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809023 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809029 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809036 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809040 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809047 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809053 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809057 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809060 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809063 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809066 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809069 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809072 2579 flags.go:64] FLAG: --cgroup-root="" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809075 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809078 2579 flags.go:64] FLAG: --client-ca-file="" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809081 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809084 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809088 2579 flags.go:64] FLAG: --cluster-domain="" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809091 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 19:29:25.813354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809094 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809097 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809101 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809104 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809107 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809110 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809113 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809116 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809120 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809123 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809126 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809129 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809133 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809136 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809138 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809142 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809144 2579 flags.go:64] FLAG: --enable-server="true" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809147 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809152 2579 flags.go:64] FLAG: --event-burst="100" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809155 2579 flags.go:64] FLAG: --event-qps="50" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809158 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809160 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809163 2579 flags.go:64] FLAG: --eviction-hard="" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809167 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 19:29:25.813913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809170 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809173 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809176 2579 flags.go:64] FLAG: --eviction-soft="" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809178 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809181 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809184 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809187 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809190 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809193 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809196 2579 flags.go:64] FLAG: --feature-gates="" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809200 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809203 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809206 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809209 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809212 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809215 2579 flags.go:64] FLAG: --help="false" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809218 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-82.ec2.internal" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809222 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809224 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809227 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809231 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809235 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809238 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 19:29:25.814484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809241 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809243 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809246 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809249 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809252 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809254 2579 flags.go:64] FLAG: --kube-reserved="" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809257 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809260 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809263 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809266 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809269 2579 flags.go:64] FLAG: --lock-file="" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809271 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809274 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809277 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809290 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809293 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809296 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809299 2579 flags.go:64] FLAG: --logging-format="text" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809302 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809305 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809308 2579 flags.go:64] FLAG: --manifest-url="" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809311 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809315 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809318 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809321 2579 flags.go:64] FLAG: --max-pods="110" Sep 11 19:29:25.815080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809324 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809327 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809330 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809333 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809336 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809338 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809341 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809348 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809351 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809354 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809357 2579 flags.go:64] FLAG: --pod-cidr="" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809360 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809366 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809369 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809372 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809375 2579 flags.go:64] FLAG: --port="10250" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809378 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809381 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a5b1471e076847f4" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809384 2579 flags.go:64] FLAG: --qos-reserved="" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809387 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809389 2579 flags.go:64] FLAG: --register-node="true" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809392 2579 flags.go:64] FLAG: --register-with-taints="" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809400 2579 flags.go:64] FLAG: --registry-burst="10" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809403 2579 flags.go:64] FLAG: --registry-qps="5" Sep 11 19:29:25.815692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809406 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809409 2579 flags.go:64] FLAG: --reserved-memory="" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809413 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809416 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809419 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809422 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809424 2579 flags.go:64] FLAG: --runonce="false" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809427 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809430 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809433 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809436 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809438 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809441 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809444 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809447 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809450 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809453 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809455 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809459 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809462 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809465 2579 flags.go:64] FLAG: --system-cgroups="" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809467 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809473 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809476 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809479 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 19:29:25.816350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809483 2579 flags.go:64] FLAG: --tls-min-version="" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809486 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809488 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809491 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809494 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809497 2579 flags.go:64] FLAG: --v="2" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809501 2579 flags.go:64] FLAG: --version="false" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809508 2579 flags.go:64] FLAG: --vmodule="" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809512 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.809515 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809645 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809650 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809653 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809656 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809659 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809662 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809665 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809667 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809670 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809673 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809676 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809678 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:29:25.816976 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809681 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809683 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809686 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809689 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809691 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809694 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809697 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809700 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809702 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809705 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809708 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809710 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809713 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809715 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809718 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809720 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809723 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809727 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809730 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809733 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:29:25.817509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809735 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809738 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809741 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809743 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809745 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809748 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809751 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809754 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809757 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809759 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809762 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809766 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809770 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809772 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809775 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809778 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809780 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809783 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809786 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:29:25.818013 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809788 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809791 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809793 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809796 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809799 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809801 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809804 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809807 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809809 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809812 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809817 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809820 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809825 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809829 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809832 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809837 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809840 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809843 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809847 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809850 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:29:25.818479 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809853 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809856 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809858 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809861 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809864 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809866 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809869 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809871 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809874 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809878 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809881 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809884 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809887 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809890 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809893 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809896 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809899 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809903 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809906 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:29:25.818984 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809909 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809912 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809914 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809918 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809921 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809924 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809927 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809929 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.809932 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.810496 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.817540 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.817556 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.818884 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.818892 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818949 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:29:25.819457 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818969 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818973 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818976 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818979 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818982 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818985 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818988 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818991 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818994 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.818997 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819000 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819002 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819005 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819008 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819010 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819013 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819015 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819018 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819020 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819023 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:29:25.819866 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819025 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819028 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819030 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819033 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819035 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819038 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819040 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819043 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819053 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819057 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819060 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819064 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819067 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819070 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819073 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819076 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819080 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819083 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819088 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:29:25.820373 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819091 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819094 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819097 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819100 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819103 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819106 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819108 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819111 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819113 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819117 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819121 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819124 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819127 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819130 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819133 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819136 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819139 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819141 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819144 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:29:25.820830 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819157 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819160 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819163 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819167 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819169 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819172 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819174 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819177 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819179 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819182 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819184 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819187 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819189 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819192 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819195 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819197 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819199 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819202 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819205 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819207 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:29:25.821315 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819210 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819212 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819215 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819217 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819220 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819222 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819224 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819227 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819229 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819232 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819234 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819237 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819239 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819243 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819245 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819248 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819251 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819254 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819256 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:29:25.821797 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819259 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.819264 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819376 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819380 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819383 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819386 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819390 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819393 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819396 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819399 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819401 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819404 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819406 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819409 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819411 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:29:25.822281 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819415 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819417 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819419 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819422 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819425 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819427 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819430 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819432 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819435 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819437 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819440 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819443 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819446 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819448 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819451 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819454 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819457 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819459 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819462 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819465 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:29:25.822658 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819467 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819470 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819472 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819475 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819477 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819480 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819482 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819485 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819487 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819489 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819492 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819495 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819497 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819500 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819503 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819505 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819508 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819510 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819513 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819515 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:29:25.823170 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819518 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819521 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819523 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819526 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819530 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819534 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819536 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819539 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819542 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819545 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819547 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819550 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819552 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819554 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819557 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819559 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819562 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819564 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819566 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:29:25.823645 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819569 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819571 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819574 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819577 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819579 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819582 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819584 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819587 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819589 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819592 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819594 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819597 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819599 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819603 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819606 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819609 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819612 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819615 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819617 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:29:25.824171 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819620 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819623 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819625 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819629 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819632 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819635 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819638 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:25.819641 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.819646 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.819752 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.821673 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.822484 2579 server.go:1019] "Starting client certificate rotation" Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.822590 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 19:29:25.824781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.822619 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 19:29:25.825423 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.825409 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 19:29:25.826661 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.826642 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 19:29:25.840001 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.839981 2579 log.go:25] "Validated CRI v1 runtime API" Sep 11 19:29:25.843108 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.843093 2579 log.go:25] "Validated CRI v1 image API" Sep 11 19:29:25.843997 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.843983 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 19:29:25.847837 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.847813 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8eeb72b9-d6f4-4f44-9843-37c63af2b968:/dev/nvme0n1p4 f2a455fc-11fc-4ff5-b042-f1ebc3efb2bb:/dev/nvme0n1p3] Sep 11 19:29:25.847893 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.847838 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 19:29:25.853070 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.853047 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 19:29:25.854943 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.854842 2579 manager.go:216] Machine: {Timestamp:2026-09-11 19:29:25.853373482 +0000 UTC m=+0.363552617 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3016879 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec234190b03504dcbb1cb8bd02157fb5 SystemUUID:ec234190-b035-04dc-bb1c-b8bd02157fb5 BootID:63be9303-3793-4fe7-928d-3eaab7dcc205 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a9:94:89:34:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a9:94:89:34:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:98:8f:68:67:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 19:29:25.854943 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.854939 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 19:29:25.855085 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.855070 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 19:29:25.858229 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.858206 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 19:29:25.858368 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.858234 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 19:29:25.859108 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.859098 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 19:29:25.859143 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.859111 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 19:29:25.859143 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.859119 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 19:29:25.859143 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.859131 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 19:29:25.859926 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.859916 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 19:29:25.863621 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.863610 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 19:29:25.863725 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.863717 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 19:29:25.863755 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.863748 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 11 19:29:25.863783 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.863756 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 19:29:25.863783 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.863774 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 19:29:25.863783 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.863784 2579 kubelet.go:398] "Adding apiserver pod source" Sep 11 19:29:25.863871 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.863800 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 19:29:25.864194 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.864181 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 19:29:25.864243 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.864201 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 19:29:25.866520 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.866501 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 19:29:25.868927 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.868906 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 19:29:25.869041 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.868949 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 19:29:25.870590 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870574 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 19:29:25.870659 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870602 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 19:29:25.870659 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870615 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 19:29:25.870659 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870624 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 19:29:25.870659 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870637 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 19:29:25.870659 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870646 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 19:29:25.870843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870666 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 19:29:25.870843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870675 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 19:29:25.870843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870686 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 19:29:25.870843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870695 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 19:29:25.870843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870714 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 19:29:25.870843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.870723 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 19:29:25.872007 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.871993 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 19:29:25.872074 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.872010 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 19:29:25.873006 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.872981 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 19:29:25.873078 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.873012 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 19:29:25.874584 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.874560 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 19:29:25.875669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.875654 2579 server.go:1268] "Started kubelet" Sep 11 19:29:25.875802 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.875738 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 19:29:25.875876 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.875786 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 19:29:25.875949 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.875935 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 19:29:25.876660 ip-10-0-138-82 systemd[1]: Started Kubernetes Kubelet. Sep 11 19:29:25.877601 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.877529 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 19:29:25.877976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.877940 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 11 19:29:25.880028 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.880008 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 19:29:25.882378 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.882357 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 19:29:25.882899 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.882884 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 19:29:25.883055 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.883033 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:25.883738 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.883714 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 19:29:25.883738 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.883716 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 19:29:25.883838 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.883742 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 19:29:25.883886 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.883877 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 11 19:29:25.883886 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.883885 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 11 19:29:25.884572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.884558 2579 factory.go:55] Registering systemd factory Sep 11 19:29:25.884638 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.884605 2579 factory.go:223] Registration of the systemd container factory successfully Sep 11 19:29:25.884676 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.884649 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vmgpw" Sep 11 19:29:25.884832 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.884819 2579 factory.go:153] Registering CRI-O factory Sep 11 19:29:25.884866 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.884834 2579 factory.go:223] Registration of the crio container factory successfully Sep 11 19:29:25.884900 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.884876 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 19:29:25.884900 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.884888 2579 factory.go:103] Registering Raw factory Sep 11 19:29:25.884970 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.884900 2579 manager.go:1196] Started watching for new ooms in manager Sep 11 19:29:25.885392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.885362 2579 manager.go:349] Starting recovery of all containers Sep 11 19:29:25.890516 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.890471 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 19:29:25.890721 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.890699 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 19:29:25.894590 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.893444 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-82.ec2.internal.18d45af64d246049 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-82.ec2.internal,UID:ip-10-0-138-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-82.ec2.internal,},FirstTimestamp:2026-09-11 19:29:25.875613769 +0000 UTC m=+0.385792905,LastTimestamp:2026-09-11 19:29:25.875613769 +0000 UTC m=+0.385792905,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-82.ec2.internal,}" Sep 11 19:29:25.898488 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.898468 2579 manager.go:354] Recovery completed Sep 11 19:29:25.898971 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.898930 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vmgpw" Sep 11 19:29:25.899890 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.899874 2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 11 19:29:25.902938 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.902923 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:29:25.904723 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.904707 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:29:25.904801 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.904738 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:29:25.904801 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.904752 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:29:25.905361 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.905343 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 19:29:25.905361 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.905361 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 19:29:25.905474 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.905368 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:29:25.905474 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.905378 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 19:29:25.906042 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.906025 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:29:25.906126 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.906057 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:29:25.906126 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.906071 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:29:25.906509 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.906439 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-82.ec2.internal.18d45af64ee0867e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-82.ec2.internal,UID:ip-10-0-138-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-82.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-82.ec2.internal,},FirstTimestamp:2026-09-11 19:29:25.904721534 +0000 UTC m=+0.414900670,LastTimestamp:2026-09-11 19:29:25.904721534 +0000 UTC m=+0.414900670,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-82.ec2.internal,}" Sep 11 19:29:25.907262 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.907249 2579 policy_none.go:49] "None policy: Start" Sep 11 19:29:25.907324 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.907268 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 19:29:25.907324 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.907280 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 19:29:25.909265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.909219 2579 policy_none.go:47] "Start" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.945564 2579 manager.go:337] "Starting Device Plugin manager" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.945785 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.945797 2579 server.go:87] "Starting device plugin registration server" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.946090 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.946101 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.946200 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.946291 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:25.946302 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.946977 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 19:29:25.961409 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:25.947033 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.008426 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.008384 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 19:29:26.009731 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.009708 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 19:29:26.009835 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.009760 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 19:29:26.009835 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.009784 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 19:29:26.009927 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.009866 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 19:29:26.012598 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.012580 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:29:26.046662 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.046640 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:29:26.047641 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.047625 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:29:26.047717 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.047659 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:29:26.047717 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.047671 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:29:26.047717 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.047694 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.056067 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.056004 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.056067 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.056028 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-82.ec2.internal\": node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.067688 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.067665 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.110872 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.110843 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal"] Sep 11 19:29:26.110949 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.110918 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:29:26.112679 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.112655 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:29:26.112785 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.112691 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:29:26.112785 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.112702 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:29:26.115068 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.115055 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:29:26.115200 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.115185 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.115247 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.115216 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:29:26.119305 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.119288 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:29:26.119392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.119323 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:29:26.119392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.119333 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:29:26.119392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.119294 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:29:26.119485 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.119394 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:29:26.119485 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.119407 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:29:26.121501 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.121484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.121576 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.121515 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:29:26.122231 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.122212 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:29:26.122299 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.122249 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:29:26.122299 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.122265 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:29:26.146750 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.146723 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-82.ec2.internal\" not found" node="ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.150184 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.150167 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-82.ec2.internal\" not found" node="ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.167899 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.167877 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.186532 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.186508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c02aa10d4821f669b8badb4704f6fe9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"0c02aa10d4821f669b8badb4704f6fe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.186598 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.186535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c02aa10d4821f669b8badb4704f6fe9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"0c02aa10d4821f669b8badb4704f6fe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.186598 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.186550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/110b986e19c57c6926c4a3f9fb96dca0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-82.ec2.internal\" (UID: \"110b986e19c57c6926c4a3f9fb96dca0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.268266 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.268230 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.287679 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.287650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c02aa10d4821f669b8badb4704f6fe9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"0c02aa10d4821f669b8badb4704f6fe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.287750 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.287722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c02aa10d4821f669b8badb4704f6fe9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"0c02aa10d4821f669b8badb4704f6fe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.287750 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.287663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c02aa10d4821f669b8badb4704f6fe9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"0c02aa10d4821f669b8badb4704f6fe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.287829 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.287769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/110b986e19c57c6926c4a3f9fb96dca0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-82.ec2.internal\" (UID: \"110b986e19c57c6926c4a3f9fb96dca0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.287829 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.287747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/110b986e19c57c6926c4a3f9fb96dca0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-82.ec2.internal\" (UID: \"110b986e19c57c6926c4a3f9fb96dca0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.287829 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.287787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c02aa10d4821f669b8badb4704f6fe9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"0c02aa10d4821f669b8badb4704f6fe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.369177 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.369115 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.449677 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.449633 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.452808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.452786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" Sep 11 19:29:26.469268 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.469242 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.569364 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.569327 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.669995 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.669910 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.770545 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.770514 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.822196 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.822170 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 19:29:26.822766 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.822308 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 19:29:26.870607 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.870575 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:26.903012 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.902983 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 19:24:27 +0000 UTC" deadline="2028-03-11 05:01:03.489020132 +0000 UTC" Sep 11 19:29:26.903012 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:26.903011 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13113h31m36.586011712s" Sep 11 19:29:26.971333 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:26.971290 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:27.071973 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:27.071922 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:27.111870 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:27.111839 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110b986e19c57c6926c4a3f9fb96dca0.slice/crio-626eb5c8889b66562ed05d34eb35e5b62b98b4e3e1857ac658d00846b0c78f81 WatchSource:0}: Error finding container 626eb5c8889b66562ed05d34eb35e5b62b98b4e3e1857ac658d00846b0c78f81: Status 404 returned error can't find the container with id 626eb5c8889b66562ed05d34eb35e5b62b98b4e3e1857ac658d00846b0c78f81 Sep 11 19:29:27.112092 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:27.112077 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c02aa10d4821f669b8badb4704f6fe9.slice/crio-e6d014344675a16e4b62a66118beb9fdba0998773f1a6bec072c6d039af10217 WatchSource:0}: Error finding container e6d014344675a16e4b62a66118beb9fdba0998773f1a6bec072c6d039af10217: Status 404 returned error can't find the container with id e6d014344675a16e4b62a66118beb9fdba0998773f1a6bec072c6d039af10217 Sep 11 19:29:27.116710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:27.116693 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:29:27.172127 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:27.172099 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found" Sep 11 19:29:27.172127 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:27.172105 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:29:27.183141 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:27.183110 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" Sep 11 19:29:27.193647 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:27.193624 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 19:29:27.195307 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:27.195293 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" Sep 11 19:29:27.204916 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:27.204894 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 19:29:28.756175 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:28.756146 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:29:29.206370 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.206341 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:29:29.241738 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.241709 2579 apiserver.go:52] "Watching apiserver" Sep 11 19:29:29.250785 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.250757 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 19:29:29.252083 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.252055 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-k6shn","openshift-network-diagnostics/network-check-target-rz9q9","openshift-network-operator/iptables-alerter-2xfph","kube-system/konnectivity-agent-24lzg","kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal","openshift-dns/node-resolver-w4bpk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal","openshift-multus/multus-additional-cni-plugins-lvxpr","openshift-ovn-kubernetes/ovnkube-node-z7ws8","kube-system/global-pull-secret-syncer-2rwkf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp","openshift-cluster-node-tuning-operator/tuned-vvrht","openshift-image-registry/node-ca-mtcmq","openshift-multus/multus-25zds"] Sep 11 19:29:29.254460 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.254435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.254573 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.254506 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:29.256664 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.256647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.258967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.258798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.259221 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.259203 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xcv6q\"" Sep 11 19:29:29.259888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.259413 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 19:29:29.259888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.259422 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 19:29:29.259888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.259435 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 19:29:29.259888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.259549 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 19:29:29.261572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.261548 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 19:29:29.261663 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.261643 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 19:29:29.261747 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.261732 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rjgk5\"" Sep 11 19:29:29.261904 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.261883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:29.261992 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.261936 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 19:29:29.264142 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.264111 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 19:29:29.267957 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.267931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25zds" Sep 11 19:29:29.268536 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.268517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 19:29:29.269122 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.268865 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rk6ll\"" Sep 11 19:29:29.271672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.270583 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 19:29:29.271672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.270788 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 19:29:29.271672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.270849 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 19:29:29.271672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.271084 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 19:29:29.271672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.271165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.271672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.271456 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dvpst\"" Sep 11 19:29:29.273251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.273233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.273342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.273282 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 19:29:29.273817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.273798 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 19:29:29.274048 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52905ff5-215f-40aa-bb52-9e786b733946-agent-certs\") pod \"konnectivity-agent-24lzg\" (UID: \"52905ff5-215f-40aa-bb52-9e786b733946\") " pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:29.274117 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.274117 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtx6x\" (UniqueName: \"kubernetes.io/projected/a81633bd-9629-4cea-bce8-00eec1f691ff-kube-api-access-xtx6x\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.274117 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8ff0233-976c-4f45-8e8d-4bd98ef51440-iptables-alerter-script\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.274265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1783842-80b8-4d47-be48-86b24985f631-dbus\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.274265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.274265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-socket-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.274265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-sys-fs\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.274265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2k7mj\"" Sep 11 19:29:29.274265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52905ff5-215f-40aa-bb52-9e786b733946-konnectivity-ca\") pod \"konnectivity-agent-24lzg\" (UID: \"52905ff5-215f-40aa-bb52-9e786b733946\") " pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:29.274265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1783842-80b8-4d47-be48-86b24985f631-kubelet-config\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.274641 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-registration-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.274641 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274279 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 19:29:29.274641 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-device-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.274641 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbcp\" (UniqueName: \"kubernetes.io/projected/d8ff0233-976c-4f45-8e8d-4bd98ef51440-kube-api-access-vvbcp\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.274641 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.274641 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.274364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8ff0233-976c-4f45-8e8d-4bd98ef51440-host-slash\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.275671 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.275654 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 19:29:29.275958 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.275942 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 19:29:29.276070 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.276055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.277084 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.277062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c6fd7\"" Sep 11 19:29:29.277340 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.277324 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 19:29:29.277508 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.277362 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 19:29:29.277560 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.277510 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 19:29:29.277761 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.277739 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 19:29:29.278155 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.278139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:29.278231 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.278205 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:29.278390 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.278372 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 19:29:29.278539 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.278492 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 19:29:29.278746 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.278727 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqhgm\"" Sep 11 19:29:29.278868 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.278752 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 19:29:29.280217 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.280201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:29.280317 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.280265 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:29.282493 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.282473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.284731 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.284712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.284965 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.284939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vnpcr\"" Sep 11 19:29:29.285204 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.285189 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 19:29:29.285341 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.285195 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 19:29:29.287089 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.287071 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 19:29:29.288527 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.288506 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 19:29:29.288620 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.288529 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jz2v5\"" Sep 11 19:29:29.292455 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.292438 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cftz7" Sep 11 19:29:29.299813 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.299797 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cftz7" Sep 11 19:29:29.361658 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.361626 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 19:29:29.374586 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.374751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52905ff5-215f-40aa-bb52-9e786b733946-konnectivity-ca\") pod \"konnectivity-agent-24lzg\" (UID: \"52905ff5-215f-40aa-bb52-9e786b733946\") " pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:29.374751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-system-cni-dir\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.374751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysctl-d\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.374751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-var-lib-kubelet\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.374751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-run-netns\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-k8s-cni-cncf-io\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbcp\" (UniqueName: \"kubernetes.io/projected/d8ff0233-976c-4f45-8e8d-4bd98ef51440-kube-api-access-vvbcp\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrd2\" (UniqueName: \"kubernetes.io/projected/d380b676-3396-49b0-a514-fffbbad37502-kube-api-access-xdrd2\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-kubernetes\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-cnibin\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-ovnkube-config\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqg5\" (UniqueName: \"kubernetes.io/projected/fb818bad-1ca1-468c-a001-711138c499ef-kube-api-access-prqg5\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.375020 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.374978 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.374985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-cnibin\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.375052 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret podName:c1783842-80b8-4d47-be48-86b24985f631 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:29.875029989 +0000 UTC m=+3.008342557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret") pod "global-pull-secret-syncer-2rwkf" (UID: "c1783842-80b8-4d47-be48-86b24985f631") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-systemd\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdsc\" (UniqueName: \"kubernetes.io/projected/3408994a-6e96-4fe7-92c3-310beaf17088-kube-api-access-8tdsc\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-tmp-dir\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-systemd\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7cw\" (UniqueName: \"kubernetes.io/projected/47ffd275-74af-4c08-9970-85bdc7f3cb07-kube-api-access-7k7cw\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375207 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52905ff5-215f-40aa-bb52-9e786b733946-konnectivity-ca\") pod \"konnectivity-agent-24lzg\" (UID: \"52905ff5-215f-40aa-bb52-9e786b733946\") " pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3408994a-6e96-4fe7-92c3-310beaf17088-tmp\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.375418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-var-lib-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47ffd275-74af-4c08-9970-85bdc7f3cb07-serviceca\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-multus-certs\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-socket-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysctl-conf\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-etc-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375634 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-cni-bin\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8ff0233-976c-4f45-8e8d-4bd98ef51440-host-slash\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8ff0233-976c-4f45-8e8d-4bd98ef51440-host-slash\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtx6x\" (UniqueName: \"kubernetes.io/projected/a81633bd-9629-4cea-bce8-00eec1f691ff-kube-api-access-xtx6x\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-socket-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-cni-multus\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.375962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-cni-bin\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-sys-fs\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-socket-dir-parent\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-netns\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.375989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-ovn\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-sys-fs\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b01d3d6-8e34-4571-a00f-bb2579d8a632-cni-binary-copy\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-kubelet\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-conf-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-registration-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-device-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-lib-modules\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-slash\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-log-socket\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysconfig\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-registration-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-run\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.376734 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3408994a-6e96-4fe7-92c3-310beaf17088-etc-tuned\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-hostroot\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-device-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-etc-kubernetes\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-hosts-file\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-kubelet\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8ff0233-976c-4f45-8e8d-4bd98ef51440-iptables-alerter-script\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-cni-netd\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-ovnkube-script-lib\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-daemon-config\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1783842-80b8-4d47-be48-86b24985f631-kubelet-config\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-modprobe-d\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-sys\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-host\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-cni-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.377613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwvd\" (UniqueName: \"kubernetes.io/projected/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-kube-api-access-6pwvd\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9lb\" (UniqueName: \"kubernetes.io/projected/4b01d3d6-8e34-4571-a00f-bb2579d8a632-kube-api-access-zt9lb\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-systemd-units\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb818bad-1ca1-468c-a001-711138c499ef-ovn-node-metrics-cert\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnffp\" (UniqueName: \"kubernetes.io/projected/dcd00465-6964-4cd1-8dd5-fda2a726811a-kube-api-access-cnffp\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52905ff5-215f-40aa-bb52-9e786b733946-agent-certs\") pod \"konnectivity-agent-24lzg\" (UID: \"52905ff5-215f-40aa-bb52-9e786b733946\") " pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-system-cni-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8ff0233-976c-4f45-8e8d-4bd98ef51440-iptables-alerter-script\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-env-overrides\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ffd275-74af-4c08-9970-85bdc7f3cb07-host\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.376986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1783842-80b8-4d47-be48-86b24985f631-dbus\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.377063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-node-log\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.377139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1783842-80b8-4d47-be48-86b24985f631-dbus\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.377188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1783842-80b8-4d47-be48-86b24985f631-kubelet-config\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.377367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a81633bd-9629-4cea-bce8-00eec1f691ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.378417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.377380 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 19:29:29.381669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.381644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52905ff5-215f-40aa-bb52-9e786b733946-agent-certs\") pod \"konnectivity-agent-24lzg\" (UID: \"52905ff5-215f-40aa-bb52-9e786b733946\") " pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:29.384307 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.384090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbcp\" (UniqueName: \"kubernetes.io/projected/d8ff0233-976c-4f45-8e8d-4bd98ef51440-kube-api-access-vvbcp\") pod \"iptables-alerter-2xfph\" (UID: \"d8ff0233-976c-4f45-8e8d-4bd98ef51440\") " pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.384307 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.384242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtx6x\" (UniqueName: \"kubernetes.io/projected/a81633bd-9629-4cea-bce8-00eec1f691ff-kube-api-access-xtx6x\") pod \"aws-ebs-csi-driver-node-nbgmp\" (UID: \"a81633bd-9629-4cea-bce8-00eec1f691ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.391547 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.391496 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" event={"ID":"0c02aa10d4821f669b8badb4704f6fe9","Type":"ContainerStarted","Data":"e6d014344675a16e4b62a66118beb9fdba0998773f1a6bec072c6d039af10217"} Sep 11 19:29:29.392771 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.392746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" event={"ID":"110b986e19c57c6926c4a3f9fb96dca0","Type":"ContainerStarted","Data":"626eb5c8889b66562ed05d34eb35e5b62b98b4e3e1857ac658d00846b0c78f81"} Sep 11 19:29:29.477751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-kubernetes\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.477751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-cnibin\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.477751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-ovnkube-config\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.477751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqg5\" (UniqueName: \"kubernetes.io/projected/fb818bad-1ca1-468c-a001-711138c499ef-kube-api-access-prqg5\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-cnibin\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-systemd\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-cnibin\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdsc\" (UniqueName: \"kubernetes.io/projected/3408994a-6e96-4fe7-92c3-310beaf17088-kube-api-access-8tdsc\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-kubernetes\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-cnibin\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-tmp-dir\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-systemd\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-systemd\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7cw\" (UniqueName: \"kubernetes.io/projected/47ffd275-74af-4c08-9970-85bdc7f3cb07-kube-api-access-7k7cw\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.477997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-systemd\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3408994a-6e96-4fe7-92c3-310beaf17088-tmp\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-var-lib-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47ffd275-74af-4c08-9970-85bdc7f3cb07-serviceca\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-tmp-dir\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-multus-certs\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-var-lib-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysctl-conf\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-etc-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.478256 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.478319 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:29:29.978299139 +0000 UTC m=+3.111611708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-ovnkube-config\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478227 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-multus-certs\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-cni-bin\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-cni-bin\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.478810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-cni-multus\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-cni-bin\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysctl-conf\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-socket-dir-parent\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478481 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-etc-openvswitch\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478443 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-cni-multus\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-netns\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-ovn\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-cni-bin\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b01d3d6-8e34-4571-a00f-bb2579d8a632-cni-binary-copy\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47ffd275-74af-4c08-9970-85bdc7f3cb07-serviceca\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-kubelet\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-netns\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-conf-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-socket-dir-parent\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-lib-modules\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-slash\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478636 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-var-lib-kubelet\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.479882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-slash\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478711 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-conf-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-lib-modules\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-run-ovn\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-log-socket\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysconfig\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-run\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3408994a-6e96-4fe7-92c3-310beaf17088-etc-tuned\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-hostroot\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysconfig\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-etc-kubernetes\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-etc-kubernetes\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-hosts-file\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-kubelet\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-run\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.478992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-cni-netd\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.480710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-ovnkube-script-lib\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479049 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-hostroot\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-daemon-config\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-kubelet\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-cni-netd\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-modprobe-d\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479113 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-log-socket\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-sys\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-hosts-file\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b01d3d6-8e34-4571-a00f-bb2579d8a632-cni-binary-copy\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-host\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479177 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-modprobe-d\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-cni-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwvd\" (UniqueName: \"kubernetes.io/projected/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-kube-api-access-6pwvd\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-sys\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9lb\" (UniqueName: \"kubernetes.io/projected/4b01d3d6-8e34-4571-a00f-bb2579d8a632-kube-api-access-zt9lb\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-systemd-units\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb818bad-1ca1-468c-a001-711138c499ef-ovn-node-metrics-cert\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnffp\" (UniqueName: \"kubernetes.io/projected/dcd00465-6964-4cd1-8dd5-fda2a726811a-kube-api-access-cnffp\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-system-cni-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-env-overrides\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ffd275-74af-4c08-9970-85bdc7f3cb07-host\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-node-log\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-system-cni-dir\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysctl-d\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-var-lib-kubelet\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-run-netns\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-k8s-cni-cncf-io\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrd2\" (UniqueName: \"kubernetes.io/projected/d380b676-3396-49b0-a514-fffbbad37502-kube-api-access-xdrd2\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-ovnkube-script-lib\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-host\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.481931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.479792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-daemon-config\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-multus-cni-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480102 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb818bad-1ca1-468c-a001-711138c499ef-env-overrides\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d380b676-3396-49b0-a514-fffbbad37502-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-node-log\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480217 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-system-cni-dir\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47ffd275-74af-4c08-9970-85bdc7f3cb07-host\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d380b676-3396-49b0-a514-fffbbad37502-system-cni-dir\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480265 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-var-lib-kubelet\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-host-run-netns\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480280 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b01d3d6-8e34-4571-a00f-bb2579d8a632-host-run-k8s-cni-cncf-io\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480306 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb818bad-1ca1-468c-a001-711138c499ef-systemd-units\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.480349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3408994a-6e96-4fe7-92c3-310beaf17088-etc-sysctl-d\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.481100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3408994a-6e96-4fe7-92c3-310beaf17088-tmp\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.481627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3408994a-6e96-4fe7-92c3-310beaf17088-etc-tuned\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.482534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.482229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb818bad-1ca1-468c-a001-711138c499ef-ovn-node-metrics-cert\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.486982 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.486580 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:29:29.486982 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.486606 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:29:29.486982 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.486618 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mwrt6 for pod openshift-network-diagnostics/network-check-target-rz9q9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:29.486982 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.486677 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6 podName:8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e nodeName:}" failed. No retries permitted until 2026-09-11 19:29:29.986658641 +0000 UTC m=+3.119971221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mwrt6" (UniqueName: "kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6") pod "network-check-target-rz9q9" (UID: "8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:29.489624 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.489597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrd2\" (UniqueName: \"kubernetes.io/projected/d380b676-3396-49b0-a514-fffbbad37502-kube-api-access-xdrd2\") pod \"multus-additional-cni-plugins-lvxpr\" (UID: \"d380b676-3396-49b0-a514-fffbbad37502\") " pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.489928 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.489909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnffp\" (UniqueName: \"kubernetes.io/projected/dcd00465-6964-4cd1-8dd5-fda2a726811a-kube-api-access-cnffp\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:29.490393 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.490374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdsc\" (UniqueName: \"kubernetes.io/projected/3408994a-6e96-4fe7-92c3-310beaf17088-kube-api-access-8tdsc\") pod \"tuned-vvrht\" (UID: \"3408994a-6e96-4fe7-92c3-310beaf17088\") " pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.490888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.490846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqg5\" (UniqueName: \"kubernetes.io/projected/fb818bad-1ca1-468c-a001-711138c499ef-kube-api-access-prqg5\") pod \"ovnkube-node-z7ws8\" (UID: \"fb818bad-1ca1-468c-a001-711138c499ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.491089 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.491069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7cw\" (UniqueName: \"kubernetes.io/projected/47ffd275-74af-4c08-9970-85bdc7f3cb07-kube-api-access-7k7cw\") pod \"node-ca-mtcmq\" (UID: \"47ffd275-74af-4c08-9970-85bdc7f3cb07\") " pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.491859 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.491840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwvd\" (UniqueName: \"kubernetes.io/projected/dcffbee5-796f-4f5b-9a7d-0e28a4c065c2-kube-api-access-6pwvd\") pod \"node-resolver-w4bpk\" (UID: \"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2\") " pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.491943 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.491862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9lb\" (UniqueName: \"kubernetes.io/projected/4b01d3d6-8e34-4571-a00f-bb2579d8a632-kube-api-access-zt9lb\") pod \"multus-25zds\" (UID: \"4b01d3d6-8e34-4571-a00f-bb2579d8a632\") " pod="openshift-multus/multus-25zds" Sep 11 19:29:29.572791 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.572748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" Sep 11 19:29:29.582191 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.582172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2xfph" Sep 11 19:29:29.590921 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.590897 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:29.597557 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.597536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25zds" Sep 11 19:29:29.612201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.612169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w4bpk" Sep 11 19:29:29.617839 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.617804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:29.626332 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.626302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mtcmq" Sep 11 19:29:29.632882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.632865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" Sep 11 19:29:29.638399 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.638379 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvrht" Sep 11 19:29:29.774409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.774337 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:29:29.882840 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.882800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:29.883015 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.882931 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:29.883015 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.882996 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret podName:c1783842-80b8-4d47-be48-86b24985f631 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:30.882980355 +0000 UTC m=+4.016292946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret") pod "global-pull-secret-syncer-2rwkf" (UID: "c1783842-80b8-4d47-be48-86b24985f631") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:29.984180 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:29.984147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:29.984326 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.984281 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:29.984369 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:29.984350 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:29:30.984331678 +0000 UTC m=+4.117644252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:30.085478 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:30.085395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:30.085619 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:30.085539 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:29:30.085619 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:30.085555 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:29:30.085619 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:30.085565 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mwrt6 for pod openshift-network-diagnostics/network-check-target-rz9q9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:30.085619 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:30.085611 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6 podName:8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e nodeName:}" failed. No retries permitted until 2026-09-11 19:29:31.085598204 +0000 UTC m=+4.218910784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwrt6" (UniqueName: "kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6") pod "network-check-target-rz9q9" (UID: "8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:30.302903 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:30.302760 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:24:29 +0000 UTC" deadline="2028-06-25 17:16:20.754681932 +0000 UTC" Sep 11 19:29:30.302903 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:30.302840 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15669h46m50.451864453s" Sep 11 19:29:30.470544 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.470515 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81633bd_9629_4cea_bce8_00eec1f691ff.slice/crio-4ce29dd0f43bdaa616d4a030ceba46123d3f58a926cc7f55720de690f0e288d6 WatchSource:0}: Error finding container 4ce29dd0f43bdaa616d4a030ceba46123d3f58a926cc7f55720de690f0e288d6: Status 404 returned error can't find the container with id 4ce29dd0f43bdaa616d4a030ceba46123d3f58a926cc7f55720de690f0e288d6 Sep 11 19:29:30.471989 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.471962 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ff0233_976c_4f45_8e8d_4bd98ef51440.slice/crio-5d137019e494ce2116319540195437366ae2b14b9b97a43d83580486f4cc181d WatchSource:0}: Error finding container 5d137019e494ce2116319540195437366ae2b14b9b97a43d83580486f4cc181d: Status 404 returned error can't find the container with id 5d137019e494ce2116319540195437366ae2b14b9b97a43d83580486f4cc181d Sep 11 19:29:30.472376 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.472351 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb818bad_1ca1_468c_a001_711138c499ef.slice/crio-d43ed02d810e66397032a651c067509dd6aedefa7a2e9091edced78440d46de7 WatchSource:0}: Error finding container d43ed02d810e66397032a651c067509dd6aedefa7a2e9091edced78440d46de7: Status 404 returned error can't find the container with id d43ed02d810e66397032a651c067509dd6aedefa7a2e9091edced78440d46de7 Sep 11 19:29:30.475578 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.475468 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcffbee5_796f_4f5b_9a7d_0e28a4c065c2.slice/crio-0cb7ce4e76d65bd70062aacd2bbddce33261f5fa3b37d8cb60ad9d03829a0dc0 WatchSource:0}: Error finding container 0cb7ce4e76d65bd70062aacd2bbddce33261f5fa3b37d8cb60ad9d03829a0dc0: Status 404 returned error can't find the container with id 0cb7ce4e76d65bd70062aacd2bbddce33261f5fa3b37d8cb60ad9d03829a0dc0 Sep 11 19:29:30.477335 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.477213 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3408994a_6e96_4fe7_92c3_310beaf17088.slice/crio-acfd784e005568a0583e6f82e8c75b257e505c7c8caeed7706a419dc4b20b341 WatchSource:0}: Error finding container acfd784e005568a0583e6f82e8c75b257e505c7c8caeed7706a419dc4b20b341: Status 404 returned error can't find the container with id acfd784e005568a0583e6f82e8c75b257e505c7c8caeed7706a419dc4b20b341 Sep 11 19:29:30.478504 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.478046 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ffd275_74af_4c08_9970_85bdc7f3cb07.slice/crio-28b72ae8e810cf907d0afbb8705204f1a359ed0a3ce4bde84f6d2554720038b6 WatchSource:0}: Error finding container 28b72ae8e810cf907d0afbb8705204f1a359ed0a3ce4bde84f6d2554720038b6: Status 404 returned error can't find the container with id 28b72ae8e810cf907d0afbb8705204f1a359ed0a3ce4bde84f6d2554720038b6 Sep 11 19:29:30.479007 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.478974 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b01d3d6_8e34_4571_a00f_bb2579d8a632.slice/crio-1e936ecb95b2a48a25bf109e43a4ee3b733e416c914a97e459352fce8bd64a66 WatchSource:0}: Error finding container 1e936ecb95b2a48a25bf109e43a4ee3b733e416c914a97e459352fce8bd64a66: Status 404 returned error can't find the container with id 1e936ecb95b2a48a25bf109e43a4ee3b733e416c914a97e459352fce8bd64a66 Sep 11 19:29:30.480239 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.480221 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52905ff5_215f_40aa_bb52_9e786b733946.slice/crio-fa64ce3ccc537fa9c63f5391201ee66211b8aa07dd7505e1badfe74c98f1b488 WatchSource:0}: Error finding container fa64ce3ccc537fa9c63f5391201ee66211b8aa07dd7505e1badfe74c98f1b488: Status 404 returned error can't find the container with id fa64ce3ccc537fa9c63f5391201ee66211b8aa07dd7505e1badfe74c98f1b488 Sep 11 19:29:30.482266 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:29:30.482241 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd380b676_3396_49b0_a514_fffbbad37502.slice/crio-6678380f24e6da2c31d243ae701e9742ff5f9bd9295256289e8c0fe8ce04bfdb WatchSource:0}: Error finding container 6678380f24e6da2c31d243ae701e9742ff5f9bd9295256289e8c0fe8ce04bfdb: Status 404 returned error can't find the container with id 6678380f24e6da2c31d243ae701e9742ff5f9bd9295256289e8c0fe8ce04bfdb Sep 11 19:29:30.891049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:30.890857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:30.891249 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:30.891079 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:30.891249 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:30.891159 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret podName:c1783842-80b8-4d47-be48-86b24985f631 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:32.891138524 +0000 UTC m=+6.024451102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret") pod "global-pull-secret-syncer-2rwkf" (UID: "c1783842-80b8-4d47-be48-86b24985f631") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:30.991796 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:30.991758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:30.991966 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:30.991946 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:30.992042 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:30.992030 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:29:32.992003575 +0000 UTC m=+6.125316149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:31.093743 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.093084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:31.093743 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:31.093316 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:29:31.093743 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:31.093340 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:29:31.093743 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:31.093353 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mwrt6 for pod openshift-network-diagnostics/network-check-target-rz9q9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:31.093743 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:31.093409 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6 podName:8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e nodeName:}" failed. No retries permitted until 2026-09-11 19:29:33.093391954 +0000 UTC m=+6.226704535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwrt6" (UniqueName: "kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6") pod "network-check-target-rz9q9" (UID: "8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:31.303896 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.303794 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:24:29 +0000 UTC" deadline="2028-06-08 20:23:10.938239564 +0000 UTC" Sep 11 19:29:31.303896 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.303853 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15264h53m39.634390816s" Sep 11 19:29:31.390078 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.390047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:31.390259 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:31.390170 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:31.390591 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.390564 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:31.390686 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:31.390668 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:31.390751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.390735 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:31.391675 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:31.390811 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:31.409683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.408863 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25zds" event={"ID":"4b01d3d6-8e34-4571-a00f-bb2579d8a632","Type":"ContainerStarted","Data":"1e936ecb95b2a48a25bf109e43a4ee3b733e416c914a97e459352fce8bd64a66"} Sep 11 19:29:31.420287 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.420177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w4bpk" event={"ID":"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2","Type":"ContainerStarted","Data":"0cb7ce4e76d65bd70062aacd2bbddce33261f5fa3b37d8cb60ad9d03829a0dc0"} Sep 11 19:29:31.429518 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.429370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"d43ed02d810e66397032a651c067509dd6aedefa7a2e9091edced78440d46de7"} Sep 11 19:29:31.451558 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.451522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" event={"ID":"d380b676-3396-49b0-a514-fffbbad37502","Type":"ContainerStarted","Data":"6678380f24e6da2c31d243ae701e9742ff5f9bd9295256289e8c0fe8ce04bfdb"} Sep 11 19:29:31.457803 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.457771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mtcmq" event={"ID":"47ffd275-74af-4c08-9970-85bdc7f3cb07","Type":"ContainerStarted","Data":"28b72ae8e810cf907d0afbb8705204f1a359ed0a3ce4bde84f6d2554720038b6"} Sep 11 19:29:31.465762 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.465735 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvrht" event={"ID":"3408994a-6e96-4fe7-92c3-310beaf17088","Type":"ContainerStarted","Data":"acfd784e005568a0583e6f82e8c75b257e505c7c8caeed7706a419dc4b20b341"} Sep 11 19:29:31.469369 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.469340 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2xfph" event={"ID":"d8ff0233-976c-4f45-8e8d-4bd98ef51440","Type":"ContainerStarted","Data":"5d137019e494ce2116319540195437366ae2b14b9b97a43d83580486f4cc181d"} Sep 11 19:29:31.475128 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.475103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" event={"ID":"a81633bd-9629-4cea-bce8-00eec1f691ff","Type":"ContainerStarted","Data":"4ce29dd0f43bdaa616d4a030ceba46123d3f58a926cc7f55720de690f0e288d6"} Sep 11 19:29:31.487301 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.487269 2579 generic.go:356] "Generic (PLEG): container finished" podID="0c02aa10d4821f669b8badb4704f6fe9" containerID="568ef3789818959590d242a6ac3a495cf53a41640f8677fa3e36fceb22826930" exitCode=0 Sep 11 19:29:31.487420 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.487353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" event={"ID":"0c02aa10d4821f669b8badb4704f6fe9","Type":"ContainerDied","Data":"568ef3789818959590d242a6ac3a495cf53a41640f8677fa3e36fceb22826930"} Sep 11 19:29:31.492493 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.492122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" event={"ID":"110b986e19c57c6926c4a3f9fb96dca0","Type":"ContainerStarted","Data":"5993560ab1440fff5cac415e610ed62a4fa01acb1fdef42e9d8f31495620bb66"} Sep 11 19:29:31.501673 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.501602 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24lzg" event={"ID":"52905ff5-215f-40aa-bb52-9e786b733946","Type":"ContainerStarted","Data":"fa64ce3ccc537fa9c63f5391201ee66211b8aa07dd7505e1badfe74c98f1b488"} Sep 11 19:29:31.520317 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:31.520248 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" podStartSLOduration=3.520226015 podStartE2EDuration="3.520226015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:31.519967227 +0000 UTC m=+4.653279817" watchObservedRunningTime="2026-09-11 19:29:31.520226015 +0000 UTC m=+4.653538606" Sep 11 19:29:32.523874 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:32.520939 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" event={"ID":"0c02aa10d4821f669b8badb4704f6fe9","Type":"ContainerStarted","Data":"66b01301e84e9e7eafccb0f50930e794ec69dcd1dcfe5ec8d6a3e9436138212f"} Sep 11 19:29:32.907801 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:32.907710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:32.907993 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:32.907888 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:32.907993 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:32.907954 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret podName:c1783842-80b8-4d47-be48-86b24985f631 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:36.907936073 +0000 UTC m=+10.041248645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret") pod "global-pull-secret-syncer-2rwkf" (UID: "c1783842-80b8-4d47-be48-86b24985f631") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:33.008358 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:33.008318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:33.008549 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.008477 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:33.008620 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.008550 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:29:37.008529496 +0000 UTC m=+10.141842066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:33.108951 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:33.108914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:33.109110 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.109092 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:29:33.109149 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.109114 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:29:33.109149 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.109127 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mwrt6 for pod openshift-network-diagnostics/network-check-target-rz9q9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:33.109212 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.109185 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6 podName:8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e nodeName:}" failed. No retries permitted until 2026-09-11 19:29:37.109166079 +0000 UTC m=+10.242478661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwrt6" (UniqueName: "kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6") pod "network-check-target-rz9q9" (UID: "8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:33.387997 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:33.387920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:33.388242 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.388186 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:33.388601 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:33.388577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:33.388713 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.388692 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:33.388962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:33.388857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:33.388962 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:33.388931 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:35.387958 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:35.387922 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:35.388453 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:35.388053 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:35.388453 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:35.388223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:35.388453 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:35.388347 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:35.388453 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:35.388400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:35.388632 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:35.388601 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:36.947194 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:36.947107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:36.947674 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:36.947291 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:36.947674 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:36.947372 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret podName:c1783842-80b8-4d47-be48-86b24985f631 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:44.947350718 +0000 UTC m=+18.080663289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret") pod "global-pull-secret-syncer-2rwkf" (UID: "c1783842-80b8-4d47-be48-86b24985f631") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:37.048454 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:37.048235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:37.048454 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.048411 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:37.048666 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.048480 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:29:45.04846172 +0000 UTC m=+18.181774301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:37.149157 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:37.149117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:37.149323 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.149287 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:29:37.149323 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.149316 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:29:37.149450 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.149328 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mwrt6 for pod openshift-network-diagnostics/network-check-target-rz9q9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:37.149450 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.149374 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6 podName:8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e nodeName:}" failed. No retries permitted until 2026-09-11 19:29:45.149362621 +0000 UTC m=+18.282675200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwrt6" (UniqueName: "kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6") pod "network-check-target-rz9q9" (UID: "8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:37.387792 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:37.387714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:37.387948 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.387858 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:37.387948 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:37.387923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:37.388036 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.387986 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:37.388068 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:37.388050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:37.388157 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:37.388129 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:39.387815 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:39.387780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:39.388431 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:39.387929 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:39.388431 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:39.388277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:39.388431 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:39.388346 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:39.388431 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:39.388401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:39.388643 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:39.388462 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:41.390593 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:41.390555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:41.391057 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:41.390684 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:41.391057 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:41.390687 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:41.391057 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:41.390697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:41.391057 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:41.390801 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:41.391057 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:41.390924 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:43.387380 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:43.387344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:43.387859 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:43.387399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:43.387859 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:43.387351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:43.387859 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:43.387487 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:43.387859 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:43.387566 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:43.387859 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:43.387649 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:45.002857 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:45.002798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:45.003286 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.002968 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:45.003286 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.003053 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret podName:c1783842-80b8-4d47-be48-86b24985f631 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:01.003032031 +0000 UTC m=+34.136344608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret") pod "global-pull-secret-syncer-2rwkf" (UID: "c1783842-80b8-4d47-be48-86b24985f631") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:29:45.103399 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:45.103361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:45.103556 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.103487 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:45.103556 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.103554 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:30:01.103534415 +0000 UTC m=+34.236847004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:29:45.203836 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:45.203780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:45.203999 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.203974 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:29:45.204075 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.203999 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:29:45.204075 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.204021 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mwrt6 for pod openshift-network-diagnostics/network-check-target-rz9q9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:45.204159 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.204086 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6 podName:8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e nodeName:}" failed. No retries permitted until 2026-09-11 19:30:01.204066835 +0000 UTC m=+34.337379415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwrt6" (UniqueName: "kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6") pod "network-check-target-rz9q9" (UID: "8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:29:45.387136 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:45.387094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:45.387136 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:45.387138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:45.387378 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:45.387094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:45.387378 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.387219 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:45.387378 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.387309 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:45.387510 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:45.387393 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:47.388351 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:47.388316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:47.388787 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:47.388380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:47.388787 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:47.388504 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:47.388787 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:47.388539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:47.388787 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:47.388606 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:47.388787 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:47.388513 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:48.567568 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.567173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24lzg" event={"ID":"52905ff5-215f-40aa-bb52-9e786b733946","Type":"ContainerStarted","Data":"e63e5fbda2e7926d1855aa65961c340a054ebb91b675729a42ec0af613add8e4"} Sep 11 19:29:48.568795 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.568753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25zds" event={"ID":"4b01d3d6-8e34-4571-a00f-bb2579d8a632","Type":"ContainerStarted","Data":"d1e496a63f967cd57a0222c8e307ca67b91a69cd6452f9a4538b14e8f6031f7c"} Sep 11 19:29:48.570463 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.570441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w4bpk" event={"ID":"dcffbee5-796f-4f5b-9a7d-0e28a4c065c2","Type":"ContainerStarted","Data":"26d15724035f611cf53d7be2cdba9feeb531e71a92339a070e37de10088bccc3"} Sep 11 19:29:48.572605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.572526 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:29:48.573071 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.573049 2579 generic.go:356] "Generic (PLEG): container finished" podID="fb818bad-1ca1-468c-a001-711138c499ef" containerID="fb0774e81d2b5e7ef1c8608d49316182bf012a6b64c4e4bb5a50dfca5fd94484" exitCode=1 Sep 11 19:29:48.573158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.573116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"6cfb22958f2c2f8cf80ce45129f2f935a1006f774618a493724db4c91cadb73f"} Sep 11 19:29:48.573218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.573174 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerDied","Data":"fb0774e81d2b5e7ef1c8608d49316182bf012a6b64c4e4bb5a50dfca5fd94484"} Sep 11 19:29:48.573218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.573208 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"53b6492d92c31cdf63cd63e8ad274e7ded3e4b1bed9dcdd2927524c5435eb92b"} Sep 11 19:29:48.574714 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.574691 2579 generic.go:356] "Generic (PLEG): container finished" podID="d380b676-3396-49b0-a514-fffbbad37502" containerID="1db3ba953d1f87a0fbca0057f32b296b5fd84dc9ee3fa1e742e2067b398c233e" exitCode=0 Sep 11 19:29:48.574802 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.574765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" event={"ID":"d380b676-3396-49b0-a514-fffbbad37502","Type":"ContainerDied","Data":"1db3ba953d1f87a0fbca0057f32b296b5fd84dc9ee3fa1e742e2067b398c233e"} Sep 11 19:29:48.576645 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.576629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mtcmq" event={"ID":"47ffd275-74af-4c08-9970-85bdc7f3cb07","Type":"ContainerStarted","Data":"96e59309c8e25649a19492adc077c972972538cec4286e715911f27e2791ecd6"} Sep 11 19:29:48.578296 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.578262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvrht" event={"ID":"3408994a-6e96-4fe7-92c3-310beaf17088","Type":"ContainerStarted","Data":"0ee23c169229c72dd06fadea858a7e3c2e964d576509da09f0017a14de3a9a90"} Sep 11 19:29:48.579859 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.579838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" event={"ID":"a81633bd-9629-4cea-bce8-00eec1f691ff","Type":"ContainerStarted","Data":"21d6bb2a55eada733be77fda140f8422e8ed6809596d61ddd221cf32e61777e6"} Sep 11 19:29:48.582587 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.582546 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-24lzg" podStartSLOduration=11.418097432 podStartE2EDuration="21.582530536s" totalImagesPullingTime="10.164433104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:48.582114549 +0000 UTC m=+21.715427140" watchObservedRunningTime="2026-09-11 19:29:48.582530536 +0000 UTC m=+21.715843125" Sep 11 19:29:48.582738 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.582714 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" podStartSLOduration=20.582708014 podStartE2EDuration="20.582708014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:32.536708266 +0000 UTC m=+5.670020856" watchObservedRunningTime="2026-09-11 19:29:48.582708014 +0000 UTC m=+21.716020645" Sep 11 19:29:48.599483 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.599446 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w4bpk" podStartSLOduration=4.05178925 podStartE2EDuration="21.599436355s" totalImagesPullingTime="17.547647105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:48.599349349 +0000 UTC m=+21.732661937" watchObservedRunningTime="2026-09-11 19:29:48.599436355 +0000 UTC m=+21.732748943" Sep 11 19:29:48.634552 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.632529 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mtcmq" podStartSLOduration=4.087276684 podStartE2EDuration="21.632509043s" totalImagesPullingTime="17.545232359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:48.631712021 +0000 UTC m=+21.765024610" watchObservedRunningTime="2026-09-11 19:29:48.632509043 +0000 UTC m=+21.765821628" Sep 11 19:29:48.649565 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.649529 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vvrht" podStartSLOduration=4.102263828 podStartE2EDuration="21.649515863s" totalImagesPullingTime="17.547252035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:48.64911727 +0000 UTC m=+21.782429872" watchObservedRunningTime="2026-09-11 19:29:48.649515863 +0000 UTC m=+21.782828455" Sep 11 19:29:48.668549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:48.668473 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-25zds" podStartSLOduration=4.065348243 podStartE2EDuration="21.668458197s" totalImagesPullingTime="17.603109954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:48.667375417 +0000 UTC m=+21.800688087" watchObservedRunningTime="2026-09-11 19:29:48.668458197 +0000 UTC m=+21.801770786" Sep 11 19:29:49.387127 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.387094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:49.387319 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.387100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:49.387319 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:49.387200 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:49.387319 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:49.387278 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:49.387424 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.387342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:49.387495 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:49.387473 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:49.587477 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.587454 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:29:49.588302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.588148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"488db8794216a9fd44c003575c60c548873079097126bfe90c7fc40d1f4272d4"} Sep 11 19:29:49.588302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.588232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"21acd5e070cd2ea9674bf55eeb3e9c19af884f84b2a1701fbdfc78cf7c3bc1cf"} Sep 11 19:29:49.588302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.588263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"98931255e160455c84985d81a80b20c577e7bd095f0c3dba6adb03b09aa8b8f8"} Sep 11 19:29:49.590078 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.590048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2xfph" event={"ID":"d8ff0233-976c-4f45-8e8d-4bd98ef51440","Type":"ContainerStarted","Data":"1cdf5082a04c6d8001ef848ed9858a20a5efcd6673a89681e3fc30d55622002f"} Sep 11 19:29:49.606740 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.606682 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2xfph" podStartSLOduration=5.066447375 podStartE2EDuration="22.606660918s" totalImagesPullingTime="17.540213543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:49.606516056 +0000 UTC m=+22.739828657" watchObservedRunningTime="2026-09-11 19:29:49.606660918 +0000 UTC m=+22.739973509" Sep 11 19:29:49.638968 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:49.638943 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 19:29:50.336844 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:50.336712 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T19:29:49.638962532Z","UUID":"75348e75-4757-4910-8867-92f310eca6a1","Handler":null,"Name":"","Endpoint":""} Sep 11 19:29:50.339586 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:50.339288 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 19:29:50.339586 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:50.339583 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 19:29:50.596227 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:50.595886 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" event={"ID":"a81633bd-9629-4cea-bce8-00eec1f691ff","Type":"ContainerStarted","Data":"9b78658436d65fdf2bdbc441951eb5d746d6874a6597dddbbb702a62d8d8ce6e"} Sep 11 19:29:51.387300 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:51.387268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:51.387484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:51.387268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:51.387484 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:51.387382 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:51.387484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:51.387400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:51.387606 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:51.387477 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:51.387606 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:51.387529 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:51.602054 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:51.602026 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:29:51.602543 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:51.602478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"873600e78aa8824eaa9abc56e22206ccba41e2d3a59ae7f2eadb607513e7d483"} Sep 11 19:29:51.604837 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:51.604805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" event={"ID":"a81633bd-9629-4cea-bce8-00eec1f691ff","Type":"ContainerStarted","Data":"899ee292c0b1ae6886a11f74c5badabac0f9369bd6e6bbcaf1a1e0f40f145ceb"} Sep 11 19:29:51.623779 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:51.623731 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbgmp" podStartSLOduration=5.014461656 podStartE2EDuration="24.623718637s" totalImagesPullingTime="19.609256981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:51.623619158 +0000 UTC m=+24.756931744" watchObservedRunningTime="2026-09-11 19:29:51.623718637 +0000 UTC m=+24.757031246" Sep 11 19:29:52.823054 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:52.823010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:52.824087 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:52.824063 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:53.387647 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.387425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:53.387782 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.387431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:53.387782 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:53.387674 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:53.387909 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:53.387785 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:53.387909 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.387441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:53.387909 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:53.387898 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:53.611000 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.610967 2579 generic.go:356] "Generic (PLEG): container finished" podID="d380b676-3396-49b0-a514-fffbbad37502" containerID="3c31c8f3aef7dca41ee34017ae5f6109a4b164b2893dc34cac578474d04137ec" exitCode=0 Sep 11 19:29:53.611157 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.611056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" event={"ID":"d380b676-3396-49b0-a514-fffbbad37502","Type":"ContainerDied","Data":"3c31c8f3aef7dca41ee34017ae5f6109a4b164b2893dc34cac578474d04137ec"} Sep 11 19:29:53.614940 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.614924 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:29:53.615348 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.615325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"a6b83eb08c9adccc1975c1cdb2f210bcc9fd7a8235ec508fa2d5c789393976fc"} Sep 11 19:29:53.615485 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.615469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:53.615529 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.615512 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:53.615677 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.615656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:53.615778 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.615728 2579 scope.go:117] "RemoveContainer" containerID="fb0774e81d2b5e7ef1c8608d49316182bf012a6b64c4e4bb5a50dfca5fd94484" Sep 11 19:29:53.615967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.615949 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-24lzg" Sep 11 19:29:53.632338 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.632317 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:53.632444 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:53.632434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:54.621503 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.621348 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:29:54.621942 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.621916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" event={"ID":"fb818bad-1ca1-468c-a001-711138c499ef","Type":"ContainerStarted","Data":"c42ab7069096c53c99d73b30dca5b1b037f213742fca769ea9c3d5d400e0173e"} Sep 11 19:29:54.622035 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.621984 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 19:29:54.624275 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.624247 2579 generic.go:356] "Generic (PLEG): container finished" podID="d380b676-3396-49b0-a514-fffbbad37502" containerID="4d7a098eb1826185f977e5d4ef3e02ade89547081568bc8376c13921f1d806b3" exitCode=0 Sep 11 19:29:54.624402 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.624311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" event={"ID":"d380b676-3396-49b0-a514-fffbbad37502","Type":"ContainerDied","Data":"4d7a098eb1826185f977e5d4ef3e02ade89547081568bc8376c13921f1d806b3"} Sep 11 19:29:54.655302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.655254 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" podStartSLOduration=10.073093716 podStartE2EDuration="27.65523658s" totalImagesPullingTime="17.582142864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:54.654749703 +0000 UTC m=+27.788062341" watchObservedRunningTime="2026-09-11 19:29:54.65523658 +0000 UTC m=+27.788549169" Sep 11 19:29:54.928933 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.928818 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rz9q9"] Sep 11 19:29:54.929096 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.928943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k6shn"] Sep 11 19:29:54.929096 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.929016 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rwkf"] Sep 11 19:29:54.929203 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.929093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:54.929203 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.929131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:54.929308 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:54.929250 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:54.930489 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:54.930256 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:54.930489 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:54.930335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:54.930489 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:54.930424 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:55.008868 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:55.008803 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:29:55.628962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:55.628921 2579 generic.go:356] "Generic (PLEG): container finished" podID="d380b676-3396-49b0-a514-fffbbad37502" containerID="bd435426d4e28ddfe9aa9a96379dda1051c71b89564cc9d60c2ae1a23ad50221" exitCode=0 Sep 11 19:29:55.629671 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:55.628993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" event={"ID":"d380b676-3396-49b0-a514-fffbbad37502","Type":"ContainerDied","Data":"bd435426d4e28ddfe9aa9a96379dda1051c71b89564cc9d60c2ae1a23ad50221"} Sep 11 19:29:56.386906 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:56.386875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:56.387067 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:56.386876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:56.387067 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:56.387015 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:29:56.387067 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:56.386876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:56.387220 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:56.387069 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:56.387220 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:56.387141 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:58.387589 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:58.387384 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:29:58.388196 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:58.387426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:29:58.388196 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:58.387685 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:29:58.388196 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:29:58.387452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:29:58.388196 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:58.387762 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:29:58.388196 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:29:58.387865 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:30:00.387262 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:00.387227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:30:00.387262 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:00.387254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:30:00.387714 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:00.387254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:30:00.387714 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:00.387327 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rwkf" podUID="c1783842-80b8-4d47-be48-86b24985f631" Sep 11 19:30:00.387714 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:00.387422 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rz9q9" podUID="8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e" Sep 11 19:30:00.387714 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:00.387494 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:30:00.648351 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:00.648265 2579 generic.go:356] "Generic (PLEG): container finished" podID="d380b676-3396-49b0-a514-fffbbad37502" containerID="218bf79e71b472bf9a21a73b8da1038f373219b2b67ee25f21e9c441c666ba6b" exitCode=0 Sep 11 19:30:00.648351 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:00.648304 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" event={"ID":"d380b676-3396-49b0-a514-fffbbad37502","Type":"ContainerDied","Data":"218bf79e71b472bf9a21a73b8da1038f373219b2b67ee25f21e9c441c666ba6b"} Sep 11 19:30:01.022008 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.021915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:30:01.022290 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.022052 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:30:01.022290 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.022108 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret podName:c1783842-80b8-4d47-be48-86b24985f631 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:33.022091688 +0000 UTC m=+66.155404290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret") pod "global-pull-secret-syncer-2rwkf" (UID: "c1783842-80b8-4d47-be48-86b24985f631") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:30:01.119865 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.119816 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeReady" Sep 11 19:30:01.120009 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.119960 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 19:30:01.123092 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.123072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:30:01.123179 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.123161 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:30:01.123227 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.123219 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:30:33.123204923 +0000 UTC m=+66.256517490 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:30:01.162796 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.162766 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n7scq"] Sep 11 19:30:01.178737 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.178711 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k5nzx"] Sep 11 19:30:01.178899 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.178883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.181883 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.181864 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 19:30:01.182000 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.181863 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 19:30:01.182000 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.181915 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6smxv\"" Sep 11 19:30:01.188944 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.188927 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n7scq"] Sep 11 19:30:01.189044 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.188956 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k5nzx"] Sep 11 19:30:01.189102 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.189050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k5nzx" Sep 11 19:30:01.191675 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.191650 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 19:30:01.191776 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.191733 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dplr6\"" Sep 11 19:30:01.191776 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.191736 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 19:30:01.192023 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.192008 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 19:30:01.224082 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.224056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:30:01.224202 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.224182 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:30:01.224202 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.224195 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:30:01.224294 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.224204 2579 projected.go:196] Error preparing data for projected volume kube-api-access-mwrt6 for pod openshift-network-diagnostics/network-check-target-rz9q9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:30:01.224294 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.224254 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6 podName:8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e nodeName:}" failed. No retries permitted until 2026-09-11 19:30:33.22424279 +0000 UTC m=+66.357555370 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwrt6" (UniqueName: "kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6") pod "network-check-target-rz9q9" (UID: "8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:30:01.324726 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.324699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-config-volume\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.324939 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.324731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/939f0c9d-6cab-4c64-adcf-8f3052559d25-cert\") pod \"ingress-canary-k5nzx\" (UID: \"939f0c9d-6cab-4c64-adcf-8f3052559d25\") " pod="openshift-ingress-canary/ingress-canary-k5nzx" Sep 11 19:30:01.324939 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.324749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.324939 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.324763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-tmp-dir\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.324939 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.324856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnbqt\" (UniqueName: \"kubernetes.io/projected/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-kube-api-access-nnbqt\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.324939 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.324886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgm6\" (UniqueName: \"kubernetes.io/projected/939f0c9d-6cab-4c64-adcf-8f3052559d25-kube-api-access-fqgm6\") pod \"ingress-canary-k5nzx\" (UID: \"939f0c9d-6cab-4c64-adcf-8f3052559d25\") " pod="openshift-ingress-canary/ingress-canary-k5nzx" Sep 11 19:30:01.425696 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.425667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-config-volume\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.426251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.425710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/939f0c9d-6cab-4c64-adcf-8f3052559d25-cert\") pod \"ingress-canary-k5nzx\" (UID: \"939f0c9d-6cab-4c64-adcf-8f3052559d25\") " pod="openshift-ingress-canary/ingress-canary-k5nzx" Sep 11 19:30:01.426251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.425739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.426251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.425762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-tmp-dir\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.426251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.425788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnbqt\" (UniqueName: \"kubernetes.io/projected/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-kube-api-access-nnbqt\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.426251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.425813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgm6\" (UniqueName: \"kubernetes.io/projected/939f0c9d-6cab-4c64-adcf-8f3052559d25-kube-api-access-fqgm6\") pod \"ingress-canary-k5nzx\" (UID: \"939f0c9d-6cab-4c64-adcf-8f3052559d25\") " pod="openshift-ingress-canary/ingress-canary-k5nzx" Sep 11 19:30:01.426251 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.425879 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:30:01.426251 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.425957 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:01.925936628 +0000 UTC m=+35.059249213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:30:01.426251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.426120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-tmp-dir\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.426642 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.426366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-config-volume\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.430265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.430241 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/939f0c9d-6cab-4c64-adcf-8f3052559d25-cert\") pod \"ingress-canary-k5nzx\" (UID: \"939f0c9d-6cab-4c64-adcf-8f3052559d25\") " pod="openshift-ingress-canary/ingress-canary-k5nzx" Sep 11 19:30:01.434884 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.434865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnbqt\" (UniqueName: \"kubernetes.io/projected/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-kube-api-access-nnbqt\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.435158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.435142 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgm6\" (UniqueName: \"kubernetes.io/projected/939f0c9d-6cab-4c64-adcf-8f3052559d25-kube-api-access-fqgm6\") pod \"ingress-canary-k5nzx\" (UID: \"939f0c9d-6cab-4c64-adcf-8f3052559d25\") " pod="openshift-ingress-canary/ingress-canary-k5nzx" Sep 11 19:30:01.497292 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.497256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k5nzx" Sep 11 19:30:01.653775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.653617 2579 generic.go:356] "Generic (PLEG): container finished" podID="d380b676-3396-49b0-a514-fffbbad37502" containerID="59c7cb181965b5a4e3bad0f7f3868e88e47066f955e068ad09cfb7f37bcdeb20" exitCode=0 Sep 11 19:30:01.653775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.653696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" event={"ID":"d380b676-3396-49b0-a514-fffbbad37502","Type":"ContainerDied","Data":"59c7cb181965b5a4e3bad0f7f3868e88e47066f955e068ad09cfb7f37bcdeb20"} Sep 11 19:30:01.847455 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.847422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k5nzx"] Sep 11 19:30:01.858173 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:30:01.858135 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939f0c9d_6cab_4c64_adcf_8f3052559d25.slice/crio-2e45caaf8bfa76e74ee40d34e25d914a10ec8de8d9076574289ad9715c41e60b WatchSource:0}: Error finding container 2e45caaf8bfa76e74ee40d34e25d914a10ec8de8d9076574289ad9715c41e60b: Status 404 returned error can't find the container with id 2e45caaf8bfa76e74ee40d34e25d914a10ec8de8d9076574289ad9715c41e60b Sep 11 19:30:01.929403 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:01.929327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:01.929552 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.929480 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:30:01.929552 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:01.929547 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:02.929531946 +0000 UTC m=+36.062844513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:30:02.251845 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.251759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:02.387087 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.387053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:30:02.387087 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.387068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:30:02.387336 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.387190 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:30:02.389874 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.389851 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7w7kq\"" Sep 11 19:30:02.390009 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.389886 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 19:30:02.390201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.390187 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 19:30:02.391059 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.391037 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 19:30:02.391173 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.391075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:02.391173 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.391155 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2xtcq\"" Sep 11 19:30:02.391285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.391247 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 19:30:02.657923 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.657886 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k5nzx" event={"ID":"939f0c9d-6cab-4c64-adcf-8f3052559d25","Type":"ContainerStarted","Data":"2e45caaf8bfa76e74ee40d34e25d914a10ec8de8d9076574289ad9715c41e60b"} Sep 11 19:30:02.661195 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.661172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" event={"ID":"d380b676-3396-49b0-a514-fffbbad37502","Type":"ContainerStarted","Data":"04c35f0b817a00e8ae0b0dd2236b66897543c106d88948e4fb3ee5c679dd6e1c"} Sep 11 19:30:02.685081 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.685031 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lvxpr" podStartSLOduration=8.202770694 podStartE2EDuration="35.685013141s" totalImagesPullingTime="27.482242447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:02.683386369 +0000 UTC m=+35.816698958" watchObservedRunningTime="2026-09-11 19:30:02.685013141 +0000 UTC m=+35.818325730" Sep 11 19:30:02.806758 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.806713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:02.864307 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.864273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:02.918492 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.918381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:02.938516 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:02.938473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:02.938715 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:02.938657 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:30:02.938768 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:02.938740 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:04.938716358 +0000 UTC m=+38.072028929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:30:03.144517 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:03.144480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:04.621888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:04.621849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:04.677121 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:04.677092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:04.953945 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:04.953919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:04.954060 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:04.954042 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:30:04.954111 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:04.954102 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:08.954088637 +0000 UTC m=+42.087401204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:30:05.484651 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:05.484614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:05.586805 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:05.586755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:05.670221 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:05.670188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k5nzx" event={"ID":"939f0c9d-6cab-4c64-adcf-8f3052559d25","Type":"ContainerStarted","Data":"ec00c426a2135e569b282235a110e10b67fcab7f24bd939fac8f841ae9ea4a0f"} Sep 11 19:30:05.686278 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:05.686232 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k5nzx" podStartSLOduration=1.653124767 podStartE2EDuration="4.686217625s" totalImagesPullingTime="3.033092858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:05.685463454 +0000 UTC m=+38.818776042" watchObservedRunningTime="2026-09-11 19:30:05.686217625 +0000 UTC m=+38.819530243" Sep 11 19:30:06.017604 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:06.017570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:06.080067 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:06.080035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:06.769732 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:06.769695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:06.827193 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:06.827165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:08.981938 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:08.981890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:08.982327 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:08.982051 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:30:08.982327 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:08.982118 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:16.982102817 +0000 UTC m=+50.115415385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:30:17.035440 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:17.035406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:17.035809 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:17.035518 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:30:17.035809 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:17.035578 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:33.035564717 +0000 UTC m=+66.168877283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:30:23.178672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:23.178639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:23.233281 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:23.233251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:26.596299 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:26.596268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:26.642473 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:26.642442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7ws8" Sep 11 19:30:26.673500 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:26.673469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:26.766752 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:26.766718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:26.825117 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:26.825089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:33.047332 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.047276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:30:33.047798 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.047352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:30:33.047798 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:33.047444 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:30:33.047798 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:33.047505 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:31:05.047489332 +0000 UTC m=+98.180801903 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:30:33.050329 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.050308 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 19:30:33.061345 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.061314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1783842-80b8-4d47-be48-86b24985f631-original-pull-secret\") pod \"global-pull-secret-syncer-2rwkf\" (UID: \"c1783842-80b8-4d47-be48-86b24985f631\") " pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:30:33.148354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.148321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:30:33.151239 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.151218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 19:30:33.159551 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:33.159527 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 19:30:33.159661 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:30:33.159613 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:31:37.159589533 +0000 UTC m=+130.292902117 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : secret "metrics-daemon-secret" not found Sep 11 19:30:33.249462 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.249426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:30:33.252511 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.252491 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 19:30:33.262255 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.262238 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 19:30:33.272882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.272854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrt6\" (UniqueName: \"kubernetes.io/projected/8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e-kube-api-access-mwrt6\") pod \"network-check-target-rz9q9\" (UID: \"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e\") " pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:30:33.305866 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.305781 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2xtcq\"" Sep 11 19:30:33.307902 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.307884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rwkf" Sep 11 19:30:33.313909 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.313891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:30:33.588396 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.588308 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rz9q9"] Sep 11 19:30:33.588922 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.588900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rwkf"] Sep 11 19:30:33.591736 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:30:33.591711 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aecd9a3_8ea1_4fc1_b2ac_3205e2cc5f0e.slice/crio-2ac6e40ea33b8c123b75af29f93dec42ecff5d86d54338df7f3fd3a1267da462 WatchSource:0}: Error finding container 2ac6e40ea33b8c123b75af29f93dec42ecff5d86d54338df7f3fd3a1267da462: Status 404 returned error can't find the container with id 2ac6e40ea33b8c123b75af29f93dec42ecff5d86d54338df7f3fd3a1267da462 Sep 11 19:30:33.592422 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:30:33.592403 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1783842_80b8_4d47_be48_86b24985f631.slice/crio-094ef82268bc3b9550b0b6b778aebe64cbd22d02d7df625d27f21c9762bdcb0e WatchSource:0}: Error finding container 094ef82268bc3b9550b0b6b778aebe64cbd22d02d7df625d27f21c9762bdcb0e: Status 404 returned error can't find the container with id 094ef82268bc3b9550b0b6b778aebe64cbd22d02d7df625d27f21c9762bdcb0e Sep 11 19:30:33.735452 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.735419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rwkf" event={"ID":"c1783842-80b8-4d47-be48-86b24985f631","Type":"ContainerStarted","Data":"094ef82268bc3b9550b0b6b778aebe64cbd22d02d7df625d27f21c9762bdcb0e"} Sep 11 19:30:33.736571 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:33.736542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rz9q9" event={"ID":"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e","Type":"ContainerStarted","Data":"2ac6e40ea33b8c123b75af29f93dec42ecff5d86d54338df7f3fd3a1267da462"} Sep 11 19:30:37.879394 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:37.879357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:37.980522 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:37.980460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:30:38.752723 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:38.752690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rz9q9" event={"ID":"8aecd9a3-8ea1-4fc1-b2ac-3205e2cc5f0e","Type":"ContainerStarted","Data":"0bf6e1815db366218e078d9dadc6e2b7e29ff733a503d921918c304c0cc03bdd"} Sep 11 19:30:38.753006 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:38.752818 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:30:38.754303 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:38.754281 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rwkf" event={"ID":"c1783842-80b8-4d47-be48-86b24985f631","Type":"ContainerStarted","Data":"8380ff002b5b222c16f68d910e7786f2c730158ece21bbba862131161ffe2987"} Sep 11 19:30:38.770002 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:38.769951 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rz9q9" podStartSLOduration=67.427464543 podStartE2EDuration="1m11.769936438s" totalImagesPullingTime="4.342471895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:38.76988042 +0000 UTC m=+71.903192999" watchObservedRunningTime="2026-09-11 19:30:38.769936438 +0000 UTC m=+71.903249029" Sep 11 19:30:38.788221 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:30:38.788166 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2rwkf" podStartSLOduration=67.442244884 podStartE2EDuration="1m11.788151045s" totalImagesPullingTime="4.345906161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:38.787204159 +0000 UTC m=+71.920516751" watchObservedRunningTime="2026-09-11 19:30:38.788151045 +0000 UTC m=+71.921463633" Sep 11 19:31:02.091613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:31:02.091509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:31:02.935063 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:31:02.935024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:31:02.996352 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:31:02.996324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:31:05.067353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:31:05.067295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:31:05.067745 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:31:05.067423 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:31:05.067745 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:31:05.067479 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:32:09.067466203 +0000 UTC m=+162.200778783 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:31:09.759245 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:31:09.759212 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rz9q9" Sep 11 19:31:37.181702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:31:37.181653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:31:37.182304 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:31:37.181800 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 19:31:37.182304 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:31:37.181913 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:33:39.181892351 +0000 UTC m=+252.315204937 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : secret "metrics-daemon-secret" not found Sep 11 19:32:04.188337 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:32:04.188293 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n7scq" podUID="d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3" Sep 11 19:32:04.957322 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:32:04.957292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7scq" Sep 11 19:32:05.396970 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:32:05.396930 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:32:09.104600 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:32:09.104549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:32:09.105032 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:32:09.104718 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:32:09.105032 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:32:09.104788 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls podName:d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3 nodeName:}" failed. No retries permitted until 2026-09-11 19:34:11.104773014 +0000 UTC m=+284.238085585 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls") pod "dns-default-n7scq" (UID: "d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3") : secret "dns-default-metrics-tls" not found Sep 11 19:32:17.388610 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:32:17.388574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:32:59.049272 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:32:59.049237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:36.145744 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.145711 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt"] Sep 11 19:33:36.147527 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.147511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:36.149313 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.149287 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm"] Sep 11 19:33:36.150968 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.150947 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 19:33:36.151984 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.151961 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 19:33:36.151984 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.151964 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 19:33:36.152149 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.152013 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wgpxq\"" Sep 11 19:33:36.152226 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.152209 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62"] Sep 11 19:33:36.152316 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.152303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm" Sep 11 19:33:36.153779 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.153762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.155267 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.155249 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vvpbx\"" Sep 11 19:33:36.157356 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.157335 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 19:33:36.157438 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.157359 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 19:33:36.158151 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.158136 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 19:33:36.158208 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.158180 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 19:33:36.158208 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.158200 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-npjnh\"" Sep 11 19:33:36.167613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.167563 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt"] Sep 11 19:33:36.168701 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.168678 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm"] Sep 11 19:33:36.169344 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.169326 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62"] Sep 11 19:33:36.196898 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.196861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wk5\" (UniqueName: \"kubernetes.io/projected/a61ca5c7-07f8-4168-9d0a-2132770b4cef-kube-api-access-j2wk5\") pod \"network-check-source-6f8b768455-dm9xm\" (UID: \"a61ca5c7-07f8-4168-9d0a-2132770b4cef\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm" Sep 11 19:33:36.197066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.196918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:36.197066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.196947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2qq\" (UniqueName: \"kubernetes.io/projected/f7996815-d98c-4995-bb2d-25f561e3ce4b-kube-api-access-fm2qq\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.197066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.196967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7996815-d98c-4995-bb2d-25f561e3ce4b-config\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.197066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.197047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grh78\" (UniqueName: \"kubernetes.io/projected/a8717083-cb9e-448d-8551-98cad4b32196-kube-api-access-grh78\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:36.197196 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.197087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7996815-d98c-4995-bb2d-25f561e3ce4b-serving-cert\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.243778 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.243743 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69"] Sep 11 19:33:36.245600 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.245585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:36.248480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.248453 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 19:33:36.248633 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.248480 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ntvw8\"" Sep 11 19:33:36.249434 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.249416 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 19:33:36.249607 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.249591 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-6g9vp"] Sep 11 19:33:36.251204 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.251190 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b65645469-d2zfh"] Sep 11 19:33:36.251358 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.251338 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.253153 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.253134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.254359 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.254341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 19:33:36.254583 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.254568 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 19:33:36.254683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.254648 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 19:33:36.254837 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.254801 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4ls2t\"" Sep 11 19:33:36.254966 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.254952 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 19:33:36.257362 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.257332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 19:33:36.257741 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.257679 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 19:33:36.257898 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.257880 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5wflx\"" Sep 11 19:33:36.257968 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.257900 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 19:33:36.258205 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.258188 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69"] Sep 11 19:33:36.267222 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.267194 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 19:33:36.269450 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.269414 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-6g9vp"] Sep 11 19:33:36.270269 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.270245 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 19:33:36.270668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.270649 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b65645469-d2zfh"] Sep 11 19:33:36.298258 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlb95\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-kube-api-access-qlb95\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.298258 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/416bd977-648b-46f7-9f45-db0b63a02ac8-tmp\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.298486 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:36.298486 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-registry-certificates\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.298486 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2qq\" (UniqueName: \"kubernetes.io/projected/f7996815-d98c-4995-bb2d-25f561e3ce4b-kube-api-access-fm2qq\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.298486 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.298418 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 19:33:36.298486 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.298477 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls podName:a8717083-cb9e-448d-8551-98cad4b32196 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:36.798461373 +0000 UTC m=+249.931773966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nj8kt" (UID: "a8717083-cb9e-448d-8551-98cad4b32196") : secret "samples-operator-tls" not found Sep 11 19:33:36.298736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/671d3002-744e-4a70-84b7-804ed2df8f86-ca-trust-extracted\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.298736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-trusted-ca\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.298736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:36.298736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/416bd977-648b-46f7-9f45-db0b63a02ac8-serving-cert\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.298736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416bd977-648b-46f7-9f45-db0b63a02ac8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.298736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416bd977-648b-46f7-9f45-db0b63a02ac8-service-ca-bundle\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.298736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ll6d\" (UniqueName: \"kubernetes.io/projected/416bd977-648b-46f7-9f45-db0b63a02ac8-kube-api-access-7ll6d\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.298736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7996815-d98c-4995-bb2d-25f561e3ce4b-config\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grh78\" (UniqueName: \"kubernetes.io/projected/a8717083-cb9e-448d-8551-98cad4b32196-kube-api-access-grh78\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-image-registry-private-configuration\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-installation-pull-secrets\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7996815-d98c-4995-bb2d-25f561e3ce4b-serving-cert\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/416bd977-648b-46f7-9f45-db0b63a02ac8-snapshots\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.298974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wk5\" (UniqueName: \"kubernetes.io/projected/a61ca5c7-07f8-4168-9d0a-2132770b4cef-kube-api-access-j2wk5\") pod \"network-check-source-6f8b768455-dm9xm\" (UID: \"a61ca5c7-07f8-4168-9d0a-2132770b4cef\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.299005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-bound-sa-token\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.299158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.299032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:36.299453 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.299252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7996815-d98c-4995-bb2d-25f561e3ce4b-config\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.301189 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.301167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7996815-d98c-4995-bb2d-25f561e3ce4b-serving-cert\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.309291 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.309239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grh78\" (UniqueName: \"kubernetes.io/projected/a8717083-cb9e-448d-8551-98cad4b32196-kube-api-access-grh78\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:36.309507 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.309487 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2qq\" (UniqueName: \"kubernetes.io/projected/f7996815-d98c-4995-bb2d-25f561e3ce4b-kube-api-access-fm2qq\") pod \"service-ca-operator-cf7c776b7-z6c62\" (UID: \"f7996815-d98c-4995-bb2d-25f561e3ce4b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.310894 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.310868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wk5\" (UniqueName: \"kubernetes.io/projected/a61ca5c7-07f8-4168-9d0a-2132770b4cef-kube-api-access-j2wk5\") pod \"network-check-source-6f8b768455-dm9xm\" (UID: \"a61ca5c7-07f8-4168-9d0a-2132770b4cef\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm" Sep 11 19:33:36.400304 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-image-registry-private-configuration\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.400304 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.400304 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-installation-pull-secrets\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.400595 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.400364 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:33:36.400595 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.400386 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b65645469-d2zfh: secret "image-registry-tls" not found Sep 11 19:33:36.400595 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/416bd977-648b-46f7-9f45-db0b63a02ac8-snapshots\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.400595 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.400457 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls podName:671d3002-744e-4a70-84b7-804ed2df8f86 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:36.900439858 +0000 UTC m=+250.033752428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls") pod "image-registry-7b65645469-d2zfh" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86") : secret "image-registry-tls" not found Sep 11 19:33:36.400595 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-bound-sa-token\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.400595 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:36.400595 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlb95\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-kube-api-access-qlb95\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/416bd977-648b-46f7-9f45-db0b63a02ac8-tmp\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.400640 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-registry-certificates\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.400712 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert podName:a7e76cf0-5a58-4409-be0e-e2bcbea840ba nodeName:}" failed. No retries permitted until 2026-09-11 19:33:36.90069348 +0000 UTC m=+250.034006069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-dkt69" (UID: "a7e76cf0-5a58-4409-be0e-e2bcbea840ba") : secret "networking-console-plugin-cert" not found Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/671d3002-744e-4a70-84b7-804ed2df8f86-ca-trust-extracted\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-trusted-ca\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/416bd977-648b-46f7-9f45-db0b63a02ac8-serving-cert\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416bd977-648b-46f7-9f45-db0b63a02ac8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416bd977-648b-46f7-9f45-db0b63a02ac8-service-ca-bundle\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.400989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ll6d\" (UniqueName: \"kubernetes.io/projected/416bd977-648b-46f7-9f45-db0b63a02ac8-kube-api-access-7ll6d\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.401345 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.400996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/416bd977-648b-46f7-9f45-db0b63a02ac8-snapshots\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.401345 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.401212 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/416bd977-648b-46f7-9f45-db0b63a02ac8-tmp\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.402192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.401718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/671d3002-744e-4a70-84b7-804ed2df8f86-ca-trust-extracted\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.402192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.401845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:36.402192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.402091 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416bd977-648b-46f7-9f45-db0b63a02ac8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.402192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.402095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-registry-certificates\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.402476 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.402218 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416bd977-648b-46f7-9f45-db0b63a02ac8-service-ca-bundle\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.402476 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.402294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-trusted-ca\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.403195 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.403170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-installation-pull-secrets\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.403299 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.403170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-image-registry-private-configuration\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.403666 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.403640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/416bd977-648b-46f7-9f45-db0b63a02ac8-serving-cert\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.412412 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.412388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-bound-sa-token\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.412500 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.412438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlb95\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-kube-api-access-qlb95\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.412558 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.412541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ll6d\" (UniqueName: \"kubernetes.io/projected/416bd977-648b-46f7-9f45-db0b63a02ac8-kube-api-access-7ll6d\") pod \"insights-operator-5844b856db-6g9vp\" (UID: \"416bd977-648b-46f7-9f45-db0b63a02ac8\") " pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.463934 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.463885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm" Sep 11 19:33:36.467280 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.467018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:36.469978 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.469961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" Sep 11 19:33:36.566930 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.566901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-6g9vp" Sep 11 19:33:36.567945 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.567919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:36.742056 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.741982 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm"] Sep 11 19:33:36.744432 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.744409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62"] Sep 11 19:33:36.745461 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:33:36.745440 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61ca5c7_07f8_4168_9d0a_2132770b4cef.slice/crio-e714a9ee4543842ae997d04a2856c44704f47369ca99b6e8f8fc0a017a1fc475 WatchSource:0}: Error finding container e714a9ee4543842ae997d04a2856c44704f47369ca99b6e8f8fc0a017a1fc475: Status 404 returned error can't find the container with id e714a9ee4543842ae997d04a2856c44704f47369ca99b6e8f8fc0a017a1fc475 Sep 11 19:33:36.746790 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:33:36.746768 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7996815_d98c_4995_bb2d_25f561e3ce4b.slice/crio-b9f9236dc8f2efb71294946cc83b095cf984bbcca9c2071ec7d86181ea2a14ba WatchSource:0}: Error finding container b9f9236dc8f2efb71294946cc83b095cf984bbcca9c2071ec7d86181ea2a14ba: Status 404 returned error can't find the container with id b9f9236dc8f2efb71294946cc83b095cf984bbcca9c2071ec7d86181ea2a14ba Sep 11 19:33:36.804472 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.804438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:36.804653 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.804549 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 19:33:36.804653 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.804600 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls podName:a8717083-cb9e-448d-8551-98cad4b32196 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:37.804585573 +0000 UTC m=+250.937898140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nj8kt" (UID: "a8717083-cb9e-448d-8551-98cad4b32196") : secret "samples-operator-tls" not found Sep 11 19:33:36.835085 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.835050 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-6g9vp"] Sep 11 19:33:36.837851 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:33:36.837792 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416bd977_648b_46f7_9f45_db0b63a02ac8.slice/crio-be2d71722378dd508e58c3df343f45cb23ecd4bd96222508c3b14c720ae0bbb2 WatchSource:0}: Error finding container be2d71722378dd508e58c3df343f45cb23ecd4bd96222508c3b14c720ae0bbb2: Status 404 returned error can't find the container with id be2d71722378dd508e58c3df343f45cb23ecd4bd96222508c3b14c720ae0bbb2 Sep 11 19:33:36.904875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.904802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:36.904875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:36.904880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:36.905136 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.904971 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:33:36.905136 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.905043 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert podName:a7e76cf0-5a58-4409-be0e-e2bcbea840ba nodeName:}" failed. No retries permitted until 2026-09-11 19:33:37.905026202 +0000 UTC m=+251.038338781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-dkt69" (UID: "a7e76cf0-5a58-4409-be0e-e2bcbea840ba") : secret "networking-console-plugin-cert" not found Sep 11 19:33:36.905136 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.904974 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:33:36.905136 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.905066 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b65645469-d2zfh: secret "image-registry-tls" not found Sep 11 19:33:36.905136 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:36.905095 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls podName:671d3002-744e-4a70-84b7-804ed2df8f86 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:37.905086129 +0000 UTC m=+251.038398696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls") pod "image-registry-7b65645469-d2zfh" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86") : secret "image-registry-tls" not found Sep 11 19:33:37.176975 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:37.176935 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6g9vp" event={"ID":"416bd977-648b-46f7-9f45-db0b63a02ac8","Type":"ContainerStarted","Data":"be2d71722378dd508e58c3df343f45cb23ecd4bd96222508c3b14c720ae0bbb2"} Sep 11 19:33:37.178133 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:37.178111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" event={"ID":"f7996815-d98c-4995-bb2d-25f561e3ce4b","Type":"ContainerStarted","Data":"b9f9236dc8f2efb71294946cc83b095cf984bbcca9c2071ec7d86181ea2a14ba"} Sep 11 19:33:37.179708 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:37.179680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm" event={"ID":"a61ca5c7-07f8-4168-9d0a-2132770b4cef","Type":"ContainerStarted","Data":"4b7df2f9f31a5c2b75762108137ebf3c8ab8ff6bc0135cc0c7e5111ab2023ad2"} Sep 11 19:33:37.179841 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:37.179725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm" event={"ID":"a61ca5c7-07f8-4168-9d0a-2132770b4cef","Type":"ContainerStarted","Data":"e714a9ee4543842ae997d04a2856c44704f47369ca99b6e8f8fc0a017a1fc475"} Sep 11 19:33:37.196169 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:37.196113 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-dm9xm" podStartSLOduration=1.196085912 podStartE2EDuration="1.196085912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:37.195254305 +0000 UTC m=+250.328566895" watchObservedRunningTime="2026-09-11 19:33:37.196085912 +0000 UTC m=+250.329398505" Sep 11 19:33:37.811552 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:37.811440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:37.811755 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:37.811577 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 19:33:37.811755 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:37.811650 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls podName:a8717083-cb9e-448d-8551-98cad4b32196 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:39.81163065 +0000 UTC m=+252.944943233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nj8kt" (UID: "a8717083-cb9e-448d-8551-98cad4b32196") : secret "samples-operator-tls" not found Sep 11 19:33:37.912529 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:37.912495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:37.912720 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:37.912549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:37.912720 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:37.912703 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:33:37.912874 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:37.912759 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert podName:a7e76cf0-5a58-4409-be0e-e2bcbea840ba nodeName:}" failed. No retries permitted until 2026-09-11 19:33:39.912741404 +0000 UTC m=+253.046053984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-dkt69" (UID: "a7e76cf0-5a58-4409-be0e-e2bcbea840ba") : secret "networking-console-plugin-cert" not found Sep 11 19:33:37.912874 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:37.912843 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:33:37.912874 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:37.912852 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b65645469-d2zfh: secret "image-registry-tls" not found Sep 11 19:33:37.913035 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:37.912881 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls podName:671d3002-744e-4a70-84b7-804ed2df8f86 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:39.912873509 +0000 UTC m=+253.046186076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls") pod "image-registry-7b65645469-d2zfh" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86") : secret "image-registry-tls" not found Sep 11 19:33:38.077848 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:38.077746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:38.178580 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:38.178533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:39.225083 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:39.225038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:33:39.225549 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.225215 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 19:33:39.225549 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.225299 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs podName:dcd00465-6964-4cd1-8dd5-fda2a726811a nodeName:}" failed. No retries permitted until 2026-09-11 19:35:41.22528015 +0000 UTC m=+374.358592716 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs") pod "network-metrics-daemon-k6shn" (UID: "dcd00465-6964-4cd1-8dd5-fda2a726811a") : secret "metrics-daemon-secret" not found Sep 11 19:33:39.830568 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:39.830527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:39.830766 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.830654 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 19:33:39.830766 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.830708 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls podName:a8717083-cb9e-448d-8551-98cad4b32196 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:43.830694456 +0000 UTC m=+256.964007023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nj8kt" (UID: "a8717083-cb9e-448d-8551-98cad4b32196") : secret "samples-operator-tls" not found Sep 11 19:33:39.931987 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:39.931926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:39.931987 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:39.931992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:39.932222 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.932069 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:33:39.932222 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.932090 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b65645469-d2zfh: secret "image-registry-tls" not found Sep 11 19:33:39.932222 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.932111 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:33:39.932222 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.932157 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls podName:671d3002-744e-4a70-84b7-804ed2df8f86 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:43.932139153 +0000 UTC m=+257.065451741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls") pod "image-registry-7b65645469-d2zfh" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86") : secret "image-registry-tls" not found Sep 11 19:33:39.932222 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:39.932170 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert podName:a7e76cf0-5a58-4409-be0e-e2bcbea840ba nodeName:}" failed. No retries permitted until 2026-09-11 19:33:43.932164034 +0000 UTC m=+257.065476600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-dkt69" (UID: "a7e76cf0-5a58-4409-be0e-e2bcbea840ba") : secret "networking-console-plugin-cert" not found Sep 11 19:33:40.191500 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.191400 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6g9vp" event={"ID":"416bd977-648b-46f7-9f45-db0b63a02ac8","Type":"ContainerStarted","Data":"aba1f97708ebdd797738c7465b41a24dc2ee78641f3539918420e2cfe6a550c4"} Sep 11 19:33:40.192960 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.192926 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" event={"ID":"f7996815-d98c-4995-bb2d-25f561e3ce4b","Type":"ContainerStarted","Data":"bd01ed4e6a7d650441ff49112c7096d63d4b6ffe3b78bd20618be644a5c91628"} Sep 11 19:33:40.209525 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.209465 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-6g9vp" podStartSLOduration=1.765983208 podStartE2EDuration="4.20945069s" totalImagesPullingTime="2.443467482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:40.209054922 +0000 UTC m=+253.342367512" watchObservedRunningTime="2026-09-11 19:33:40.20945069 +0000 UTC m=+253.342763278" Sep 11 19:33:40.227804 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.227742 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" podStartSLOduration=1.69059982 podStartE2EDuration="4.227719712s" totalImagesPullingTime="2.537119892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:40.227246447 +0000 UTC m=+253.360559037" watchObservedRunningTime="2026-09-11 19:33:40.227719712 +0000 UTC m=+253.361032302" Sep 11 19:33:40.317453 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.317415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:40.374907 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.374868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:40.432927 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.432892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:40.504588 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.504477 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d2gbp"] Sep 11 19:33:40.507097 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.507073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.508341 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.508303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:40.509868 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.509818 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 19:33:40.509964 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.509849 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vrbnh\"" Sep 11 19:33:40.509964 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.509926 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 19:33:40.519690 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.519663 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d2gbp"] Sep 11 19:33:40.637769 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.637728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.638028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.637774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/50fa5062-1652-4a9c-b9ae-ee712425592e-crio-socket\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.638028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.637853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqst8\" (UniqueName: \"kubernetes.io/projected/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-api-access-pqst8\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.638028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.637907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.638028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.637936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/50fa5062-1652-4a9c-b9ae-ee712425592e-data-volume\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.739212 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.739175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.739352 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.739233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/50fa5062-1652-4a9c-b9ae-ee712425592e-crio-socket\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.739352 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.739267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqst8\" (UniqueName: \"kubernetes.io/projected/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-api-access-pqst8\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.739352 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.739303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.739352 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:40.739339 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:33:40.739541 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:40.739370 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:33:40.739541 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.739388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/50fa5062-1652-4a9c-b9ae-ee712425592e-data-volume\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.739541 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.739395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/50fa5062-1652-4a9c-b9ae-ee712425592e-crio-socket\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.739541 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:40.739405 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:41.239386519 +0000 UTC m=+254.372699103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : secret "insights-runtime-extractor-tls" not found Sep 11 19:33:40.739541 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:40.739459 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:41.239440852 +0000 UTC m=+254.372753423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : configmap "kube-rbac-proxy" not found Sep 11 19:33:40.739699 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.739640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/50fa5062-1652-4a9c-b9ae-ee712425592e-data-volume\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.750313 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.750277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqst8\" (UniqueName: \"kubernetes.io/projected/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-api-access-pqst8\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:40.771749 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:40.771645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:41.211860 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:41.211811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:41.244672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:41.244634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:41.245055 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:41.244710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:41.245055 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:41.244794 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:33:41.245055 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:41.244799 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:33:41.245055 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:41.244880 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:42.2448618 +0000 UTC m=+255.378174368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : secret "insights-runtime-extractor-tls" not found Sep 11 19:33:41.245055 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:41.244897 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:42.244888916 +0000 UTC m=+255.378201483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : configmap "kube-rbac-proxy" not found Sep 11 19:33:41.314536 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:41.314503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:41.567483 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:41.567446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:41.666435 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:41.666403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:42.253891 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:42.253843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:42.254346 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:42.253920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:42.254346 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:42.253940 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:33:42.254346 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:42.254000 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:44.253985212 +0000 UTC m=+257.387297780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : secret "insights-runtime-extractor-tls" not found Sep 11 19:33:42.254346 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:42.254006 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:33:42.254346 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:42.254046 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:44.254034605 +0000 UTC m=+257.387347203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : configmap "kube-rbac-proxy" not found Sep 11 19:33:43.091522 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.091484 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-m7gmh"] Sep 11 19:33:43.093607 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.093591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.096611 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.096584 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 19:33:43.096727 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.096626 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 19:33:43.097760 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.097742 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5z6zx\"" Sep 11 19:33:43.097905 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.097778 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 19:33:43.097905 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.097781 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 19:33:43.104052 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.104031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-m7gmh"] Sep 11 19:33:43.122847 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.122799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:43.161049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.161014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/34af7363-366d-4164-8c54-df1fa3c65745-signing-cabundle\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.161171 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.161090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/34af7363-366d-4164-8c54-df1fa3c65745-signing-key\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.161171 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.161129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkrg\" (UniqueName: \"kubernetes.io/projected/34af7363-366d-4164-8c54-df1fa3c65745-kube-api-access-zdkrg\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.224532 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.224502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:43.261497 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.261463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/34af7363-366d-4164-8c54-df1fa3c65745-signing-key\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.261900 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.261515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkrg\" (UniqueName: \"kubernetes.io/projected/34af7363-366d-4164-8c54-df1fa3c65745-kube-api-access-zdkrg\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.261900 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.261562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/34af7363-366d-4164-8c54-df1fa3c65745-signing-cabundle\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.262187 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.262170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/34af7363-366d-4164-8c54-df1fa3c65745-signing-cabundle\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.264022 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.264001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/34af7363-366d-4164-8c54-df1fa3c65745-signing-key\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.275448 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.275420 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w4bpk_dcffbee5-796f-4f5b-9a7d-0e28a4c065c2/dns-node-resolver/0.log" Sep 11 19:33:43.278045 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.278025 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkrg\" (UniqueName: \"kubernetes.io/projected/34af7363-366d-4164-8c54-df1fa3c65745-kube-api-access-zdkrg\") pod \"service-ca-568869fdc5-m7gmh\" (UID: \"34af7363-366d-4164-8c54-df1fa3c65745\") " pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.402792 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.402701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" Sep 11 19:33:43.678946 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.678859 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-m7gmh"] Sep 11 19:33:43.681572 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:33:43.681547 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34af7363_366d_4164_8c54_df1fa3c65745.slice/crio-82f6591389d4a7f7b54168c85364c73811d78ce75d94be892113e19902bbc90f WatchSource:0}: Error finding container 82f6591389d4a7f7b54168c85364c73811d78ce75d94be892113e19902bbc90f: Status 404 returned error can't find the container with id 82f6591389d4a7f7b54168c85364c73811d78ce75d94be892113e19902bbc90f Sep 11 19:33:43.867644 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.867613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:43.867849 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:43.867776 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 19:33:43.867932 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:43.867868 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls podName:a8717083-cb9e-448d-8551-98cad4b32196 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:51.867847376 +0000 UTC m=+265.001159943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nj8kt" (UID: "a8717083-cb9e-448d-8551-98cad4b32196") : secret "samples-operator-tls" not found Sep 11 19:33:43.969134 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.969031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:43.969134 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:43.969086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:43.969360 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:43.969205 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:33:43.969360 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:43.969218 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:33:43.969360 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:43.969240 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b65645469-d2zfh: secret "image-registry-tls" not found Sep 11 19:33:43.969360 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:43.969286 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert podName:a7e76cf0-5a58-4409-be0e-e2bcbea840ba nodeName:}" failed. No retries permitted until 2026-09-11 19:33:51.969267298 +0000 UTC m=+265.102579881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-dkt69" (UID: "a7e76cf0-5a58-4409-be0e-e2bcbea840ba") : secret "networking-console-plugin-cert" not found Sep 11 19:33:43.969360 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:43.969304 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls podName:671d3002-744e-4a70-84b7-804ed2df8f86 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:51.96929492 +0000 UTC m=+265.102607486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls") pod "image-registry-7b65645469-d2zfh" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86") : secret "image-registry-tls" not found Sep 11 19:33:44.064815 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.064784 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mtcmq_47ffd275-74af-4c08-9970-85bdc7f3cb07/node-ca/0.log" Sep 11 19:33:44.066904 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.066879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:44.208330 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.208302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" event={"ID":"34af7363-366d-4164-8c54-df1fa3c65745","Type":"ContainerStarted","Data":"8fdb14a6f22b230a6f4ba98e6ba47f35096ffe98d07b7baa7681f8f3645fdcaf"} Sep 11 19:33:44.208544 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.208352 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" event={"ID":"34af7363-366d-4164-8c54-df1fa3c65745","Type":"ContainerStarted","Data":"82f6591389d4a7f7b54168c85364c73811d78ce75d94be892113e19902bbc90f"} Sep 11 19:33:44.225679 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.225578 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-m7gmh" podStartSLOduration=1.225556308 podStartE2EDuration="1.225556308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:44.223898726 +0000 UTC m=+257.357211315" watchObservedRunningTime="2026-09-11 19:33:44.225556308 +0000 UTC m=+257.358868900" Sep 11 19:33:44.271841 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.271793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:44.272334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.271873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:44.272334 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:44.271925 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:33:44.272334 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:44.271974 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:48.271960417 +0000 UTC m=+261.405272984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : configmap "kube-rbac-proxy" not found Sep 11 19:33:44.272334 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:44.271974 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:33:44.272334 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:44.272042 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:48.272023838 +0000 UTC m=+261.405336408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : secret "insights-runtime-extractor-tls" not found Sep 11 19:33:44.816464 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.816422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:44.865033 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.864995 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k5nzx_939f0c9d-6cab-4c64-adcf-8f3052559d25/serve-healthcheck-canary/0.log" Sep 11 19:33:44.984675 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:44.984639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:45.086612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:45.086518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:45.236294 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:45.236243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:45.337792 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:45.337707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:45.410448 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:45.410400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:45.719136 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:45.719018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:46.320087 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:46.320054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:47.681698 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:47.681665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:47.778554 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:47.778519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:48.112003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:48.111947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:48.307135 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:48.307093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:48.307308 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:48.307210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:48.307308 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:48.307227 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:33:48.307308 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:48.307293 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:56.307273602 +0000 UTC m=+269.440586176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : configmap "kube-rbac-proxy" not found Sep 11 19:33:48.307424 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:48.307329 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:33:48.307424 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:33:48.307374 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls podName:50fa5062-1652-4a9c-b9ae-ee712425592e nodeName:}" failed. No retries permitted until 2026-09-11 19:33:56.307361754 +0000 UTC m=+269.440674335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2gbp" (UID: "50fa5062-1652-4a9c-b9ae-ee712425592e") : secret "insights-runtime-extractor-tls" not found Sep 11 19:33:48.514118 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:48.514006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:48.714368 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:48.714330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:49.116245 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:49.116210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:49.193986 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:49.193945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:49.527348 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:49.527266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:49.704038 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:49.704003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:49.910297 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:49.910262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:50.918187 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:50.918154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:50.973417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:50.973347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:51.106790 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:51.106741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:51.306578 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:51.306541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:51.721848 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:51.721734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:51.936151 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:51.936116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:51.938854 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:51.938807 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8717083-cb9e-448d-8551-98cad4b32196-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nj8kt\" (UID: \"a8717083-cb9e-448d-8551-98cad4b32196\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:52.037490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.037451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:52.037689 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.037501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:52.040204 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.040177 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7e76cf0-5a58-4409-be0e-e2bcbea840ba-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dkt69\" (UID: \"a7e76cf0-5a58-4409-be0e-e2bcbea840ba\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:52.040341 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.040178 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"image-registry-7b65645469-d2zfh\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:52.058019 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.057983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" Sep 11 19:33:52.107698 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.107506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:52.155275 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.155241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" Sep 11 19:33:52.172556 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.172526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:52.330489 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.330411 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt"] Sep 11 19:33:52.429231 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.429194 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69"] Sep 11 19:33:52.432210 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:33:52.432181 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e76cf0_5a58_4409_be0e_e2bcbea840ba.slice/crio-0811afc7cfe5e80601d245aee6ebf1845cb63e60f1018d1e426020849d9eb3b8 WatchSource:0}: Error finding container 0811afc7cfe5e80601d245aee6ebf1845cb63e60f1018d1e426020849d9eb3b8: Status 404 returned error can't find the container with id 0811afc7cfe5e80601d245aee6ebf1845cb63e60f1018d1e426020849d9eb3b8 Sep 11 19:33:52.451368 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.451340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b65645469-d2zfh"] Sep 11 19:33:52.454224 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:33:52.454201 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671d3002_744e_4a70_84b7_804ed2df8f86.slice/crio-d13c0041efb6816f1208b3a7c1de45a40086fe2b5fe444b7fab656980d49b9ed WatchSource:0}: Error finding container d13c0041efb6816f1208b3a7c1de45a40086fe2b5fe444b7fab656980d49b9ed: Status 404 returned error can't find the container with id d13c0041efb6816f1208b3a7c1de45a40086fe2b5fe444b7fab656980d49b9ed Sep 11 19:33:52.513760 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:52.513730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:53.241526 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.241480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" event={"ID":"a7e76cf0-5a58-4409-be0e-e2bcbea840ba","Type":"ContainerStarted","Data":"0811afc7cfe5e80601d245aee6ebf1845cb63e60f1018d1e426020849d9eb3b8"} Sep 11 19:33:53.243552 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.243516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" event={"ID":"671d3002-744e-4a70-84b7-804ed2df8f86","Type":"ContainerStarted","Data":"55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23"} Sep 11 19:33:53.243695 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.243567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" event={"ID":"671d3002-744e-4a70-84b7-804ed2df8f86","Type":"ContainerStarted","Data":"d13c0041efb6816f1208b3a7c1de45a40086fe2b5fe444b7fab656980d49b9ed"} Sep 11 19:33:53.243775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.243740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:33:53.245199 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.245171 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" event={"ID":"a8717083-cb9e-448d-8551-98cad4b32196","Type":"ContainerStarted","Data":"7206d63fa2bd70ec8e98c902ede403e767e311b84d2294696b07e5a41e45a5ec"} Sep 11 19:33:53.264909 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.264819 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" podStartSLOduration=17.264796494 podStartE2EDuration="17.264796494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:53.263044709 +0000 UTC m=+266.396357332" watchObservedRunningTime="2026-09-11 19:33:53.264796494 +0000 UTC m=+266.398109084" Sep 11 19:33:53.308801 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.308769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:53.512956 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.512867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:53.708007 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.707965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:53.909490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:53.909444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:54.106989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:54.106945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:54.250994 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:54.250963 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" event={"ID":"a7e76cf0-5a58-4409-be0e-e2bcbea840ba","Type":"ContainerStarted","Data":"f1263444c511e468470376aeb504051e5d39efb2c716c82bde069bb7738d0a2f"} Sep 11 19:33:54.252742 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:54.252713 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" event={"ID":"a8717083-cb9e-448d-8551-98cad4b32196","Type":"ContainerStarted","Data":"1ef0c064c266cdd880e680f7e4c205640b9c040b40bda6528742a4aaea9cfe3b"} Sep 11 19:33:54.267604 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:54.267552 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dkt69" podStartSLOduration=16.569210864 podStartE2EDuration="18.267532752s" totalImagesPullingTime="1.698321888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:54.266571119 +0000 UTC m=+267.399883707" watchObservedRunningTime="2026-09-11 19:33:54.267532752 +0000 UTC m=+267.400845343" Sep 11 19:33:54.308062 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:54.308035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:55.107455 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:55.107414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:55.258005 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:55.257971 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" event={"ID":"a8717083-cb9e-448d-8551-98cad4b32196","Type":"ContainerStarted","Data":"32188f0ebebbb548e23a13b494bd80494ae7d7662847315bbaf029ebbca84079"} Sep 11 19:33:55.277082 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:55.277027 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nj8kt" podStartSLOduration=17.510075959 podStartE2EDuration="19.277011172s" totalImagesPullingTime="1.766935213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:55.276303504 +0000 UTC m=+268.409616094" watchObservedRunningTime="2026-09-11 19:33:55.277011172 +0000 UTC m=+268.410323761" Sep 11 19:33:55.312564 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:55.312535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:55.367355 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:55.367293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:55.506709 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:55.506672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:33:56.375542 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:56.375500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:56.376057 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:56.375573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:56.376209 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:56.376191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50fa5062-1652-4a9c-b9ae-ee712425592e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:56.378056 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:56.378036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50fa5062-1652-4a9c-b9ae-ee712425592e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2gbp\" (UID: \"50fa5062-1652-4a9c-b9ae-ee712425592e\") " pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:56.416797 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:56.416759 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2gbp" Sep 11 19:33:56.686053 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:56.685980 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d2gbp"] Sep 11 19:33:56.688982 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:33:56.688951 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fa5062_1652_4a9c_b9ae_ee712425592e.slice/crio-51521350b9c35d11c2e24a219d2f3505b3fd0848a0cda150ea8732ff656a7a7b WatchSource:0}: Error finding container 51521350b9c35d11c2e24a219d2f3505b3fd0848a0cda150ea8732ff656a7a7b: Status 404 returned error can't find the container with id 51521350b9c35d11c2e24a219d2f3505b3fd0848a0cda150ea8732ff656a7a7b Sep 11 19:33:57.266318 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:57.266279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2gbp" event={"ID":"50fa5062-1652-4a9c-b9ae-ee712425592e","Type":"ContainerStarted","Data":"aacbe54a972743a6068fdecb1f625978d6c5481fd0042af01c9cd6acbd7bb44e"} Sep 11 19:33:57.266497 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:57.266353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2gbp" event={"ID":"50fa5062-1652-4a9c-b9ae-ee712425592e","Type":"ContainerStarted","Data":"51521350b9c35d11c2e24a219d2f3505b3fd0848a0cda150ea8732ff656a7a7b"} Sep 11 19:33:58.272704 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:58.272666 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2gbp" event={"ID":"50fa5062-1652-4a9c-b9ae-ee712425592e","Type":"ContainerStarted","Data":"8fba445b49a309d6a7887e275ca86477f39bb133aaeffb37894fbd063897aeed"} Sep 11 19:33:59.277881 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:59.277773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2gbp" event={"ID":"50fa5062-1652-4a9c-b9ae-ee712425592e","Type":"ContainerStarted","Data":"edf61757e871c7b968bf7c0b346f8a509a7e1147f373241cd274e87628520aae"} Sep 11 19:33:59.297864 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:33:59.297785 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d2gbp" podStartSLOduration=17.262424784 podStartE2EDuration="19.297766848s" totalImagesPullingTime="2.035342064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:33:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:59.296149115 +0000 UTC m=+272.429461705" watchObservedRunningTime="2026-09-11 19:33:59.297766848 +0000 UTC m=+272.431079437" Sep 11 19:34:02.115549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:02.115509 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b65645469-d2zfh"] Sep 11 19:34:02.230150 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:02.230108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:02.331728 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:02.331690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:05.485878 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:05.485841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:05.544612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:05.544585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:07.958256 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:07.958210 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n7scq" podUID="d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3" Sep 11 19:34:08.305445 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:08.305413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7scq" Sep 11 19:34:10.425970 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.425925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:10.530070 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.530031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:10.581491 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.581453 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h"] Sep 11 19:34:10.585503 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.585484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" Sep 11 19:34:10.588650 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.588627 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 19:34:10.588753 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.588680 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-67vb9\"" Sep 11 19:34:10.593223 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.593018 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h"] Sep 11 19:34:10.614027 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.613976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:10.685355 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.685269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/db589459-af0e-43dd-882d-d0d20d4ea99a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h9z9h\" (UID: \"db589459-af0e-43dd-882d-d0d20d4ea99a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" Sep 11 19:34:10.725118 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.725066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:10.786221 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.786176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/db589459-af0e-43dd-882d-d0d20d4ea99a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h9z9h\" (UID: \"db589459-af0e-43dd-882d-d0d20d4ea99a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" Sep 11 19:34:10.788814 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.788788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/db589459-af0e-43dd-882d-d0d20d4ea99a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h9z9h\" (UID: \"db589459-af0e-43dd-882d-d0d20d4ea99a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" Sep 11 19:34:10.896495 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:10.896459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" Sep 11 19:34:11.164912 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:11.164869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h"] Sep 11 19:34:11.168446 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:34:11.168420 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb589459_af0e_43dd_882d_d0d20d4ea99a.slice/crio-d7346c0bb8f351342e1bc3cb6f7ff81b35173989f40d4e45106a2180f4e7aab8 WatchSource:0}: Error finding container d7346c0bb8f351342e1bc3cb6f7ff81b35173989f40d4e45106a2180f4e7aab8: Status 404 returned error can't find the container with id d7346c0bb8f351342e1bc3cb6f7ff81b35173989f40d4e45106a2180f4e7aab8 Sep 11 19:34:11.189163 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:11.189136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:34:11.191593 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:11.191571 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3-metrics-tls\") pod \"dns-default-n7scq\" (UID: \"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3\") " pod="openshift-dns/dns-default-n7scq" Sep 11 19:34:11.309062 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:11.309030 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6smxv\"" Sep 11 19:34:11.315927 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:11.315896 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" event={"ID":"db589459-af0e-43dd-882d-d0d20d4ea99a","Type":"ContainerStarted","Data":"d7346c0bb8f351342e1bc3cb6f7ff81b35173989f40d4e45106a2180f4e7aab8"} Sep 11 19:34:11.316952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:11.316935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7scq" Sep 11 19:34:11.588016 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:11.587933 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n7scq"] Sep 11 19:34:11.591239 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:34:11.591211 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d9a8c5_3c75_47dd_b3ba_69d00b6397e3.slice/crio-ab5bd11e934caab2ed75575e116067727a12eea10f49564c9ae7063e87c2d35f WatchSource:0}: Error finding container ab5bd11e934caab2ed75575e116067727a12eea10f49564c9ae7063e87c2d35f: Status 404 returned error can't find the container with id ab5bd11e934caab2ed75575e116067727a12eea10f49564c9ae7063e87c2d35f Sep 11 19:34:12.123757 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:12.123720 2579 patch_prober.go:28] interesting pod/image-registry-7b65645469-d2zfh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 19:34:12.123961 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:12.123785 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" podUID="671d3002-744e-4a70-84b7-804ed2df8f86" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 19:34:12.321625 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:12.321585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7scq" event={"ID":"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3","Type":"ContainerStarted","Data":"ab5bd11e934caab2ed75575e116067727a12eea10f49564c9ae7063e87c2d35f"} Sep 11 19:34:13.328003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.327462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7scq" event={"ID":"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3","Type":"ContainerStarted","Data":"1b5b73381c5e97e56ee880ed202d475a2fe5d37dc4efcec8d2cfbbe1e0816a45"} Sep 11 19:34:13.328003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.327569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7scq" event={"ID":"d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3","Type":"ContainerStarted","Data":"a363b9d369764b9aa7738d08c08c9feab0ef9aee31a6cea78d28da1efde05d76"} Sep 11 19:34:13.328003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.327768 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n7scq" Sep 11 19:34:13.329458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.329431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" event={"ID":"db589459-af0e-43dd-882d-d0d20d4ea99a","Type":"ContainerStarted","Data":"4714a96389f16be749d24248898a86ae23cfd3478fc7255b8ea9835e8e4c4d71"} Sep 11 19:34:13.329657 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.329641 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" Sep 11 19:34:13.334447 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.334422 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" Sep 11 19:34:13.391344 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.391235 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n7scq" podStartSLOduration=250.86082628 podStartE2EDuration="4m12.39121868s" totalImagesPullingTime="1.5303924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:13.365437702 +0000 UTC m=+286.498750290" watchObservedRunningTime="2026-09-11 19:34:13.39121868 +0000 UTC m=+286.524531269" Sep 11 19:34:13.391754 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.391715 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9z9h" podStartSLOduration=1.98668105 podStartE2EDuration="3.391700911s" totalImagesPullingTime="1.405019861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:13.390701554 +0000 UTC m=+286.524014144" watchObservedRunningTime="2026-09-11 19:34:13.391700911 +0000 UTC m=+286.525013501" Sep 11 19:34:13.602623 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.602588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:13.637890 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.637860 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm"] Sep 11 19:34:13.641090 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.641072 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.644264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.644202 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 19:34:13.644399 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.644204 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 19:34:13.645135 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.645114 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 19:34:13.645227 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.645113 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 19:34:13.645227 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.645167 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 19:34:13.645227 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.645135 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8xfct\"" Sep 11 19:34:13.649583 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.649560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm"] Sep 11 19:34:13.666925 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.666896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:13.705293 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.705250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.705439 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.705303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5b52\" (UniqueName: \"kubernetes.io/projected/22885ef4-d0ba-4dfa-b293-b128087f0779-kube-api-access-s5b52\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.705439 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.705365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22885ef4-d0ba-4dfa-b293-b128087f0779-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.705439 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.705424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.806686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.806645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.806686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.806692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5b52\" (UniqueName: \"kubernetes.io/projected/22885ef4-d0ba-4dfa-b293-b128087f0779-kube-api-access-s5b52\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.806922 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.806725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22885ef4-d0ba-4dfa-b293-b128087f0779-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.806922 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.806761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.806922 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:13.806804 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 11 19:34:13.806922 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:13.806914 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-tls podName:22885ef4-d0ba-4dfa-b293-b128087f0779 nodeName:}" failed. No retries permitted until 2026-09-11 19:34:14.306890135 +0000 UTC m=+287.440202706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-f2bkm" (UID: "22885ef4-d0ba-4dfa-b293-b128087f0779") : secret "prometheus-operator-tls" not found Sep 11 19:34:13.807519 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.807494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22885ef4-d0ba-4dfa-b293-b128087f0779-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.809354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.809336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.818313 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.818285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5b52\" (UniqueName: \"kubernetes.io/projected/22885ef4-d0ba-4dfa-b293-b128087f0779-kube-api-access-s5b52\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:13.875028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:13.874994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:14.309899 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:14.309855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:14.312381 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:14.312358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22885ef4-d0ba-4dfa-b293-b128087f0779-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-f2bkm\" (UID: \"22885ef4-d0ba-4dfa-b293-b128087f0779\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:14.552002 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:14.551962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" Sep 11 19:34:14.824073 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:14.824042 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm"] Sep 11 19:34:14.826757 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:34:14.826729 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22885ef4_d0ba_4dfa_b293_b128087f0779.slice/crio-d66b7b8b73da753a84b9979a0ef461f58053031dc633a5dd3dfa1a30cbb7e5bb WatchSource:0}: Error finding container d66b7b8b73da753a84b9979a0ef461f58053031dc633a5dd3dfa1a30cbb7e5bb: Status 404 returned error can't find the container with id d66b7b8b73da753a84b9979a0ef461f58053031dc633a5dd3dfa1a30cbb7e5bb Sep 11 19:34:15.339840 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:15.339794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" event={"ID":"22885ef4-d0ba-4dfa-b293-b128087f0779","Type":"ContainerStarted","Data":"d66b7b8b73da753a84b9979a0ef461f58053031dc633a5dd3dfa1a30cbb7e5bb"} Sep 11 19:34:16.575798 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:16.575767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:17.348389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:17.348354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" event={"ID":"22885ef4-d0ba-4dfa-b293-b128087f0779","Type":"ContainerStarted","Data":"24a7304e3d4296686d0378271179f4c9cd10cd05cc913be9b64456e06bdc45bd"} Sep 11 19:34:17.348573 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:17.348421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" event={"ID":"22885ef4-d0ba-4dfa-b293-b128087f0779","Type":"ContainerStarted","Data":"0ffecf033b11bf9e60705200b4f30bfe6ebd84d47cf0dcaf26ef8cabde4612c9"} Sep 11 19:34:17.365981 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:17.365936 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-f2bkm" podStartSLOduration=2.849304295 podStartE2EDuration="4.365918916s" totalImagesPullingTime="1.516614621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:17.364800748 +0000 UTC m=+290.498113348" watchObservedRunningTime="2026-09-11 19:34:17.365918916 +0000 UTC m=+290.499231504" Sep 11 19:34:18.749959 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.749927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:18.947504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.947466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:18.965840 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.965789 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t"] Sep 11 19:34:18.969451 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.969428 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:18.972371 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.972348 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 11 19:34:18.972597 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.972355 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 11 19:34:18.972663 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.972376 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jdghj\"" Sep 11 19:34:18.980626 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.980602 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t"] Sep 11 19:34:18.989559 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.989528 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl"] Sep 11 19:34:18.993259 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.993233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:18.996351 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.996322 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-78drv"] Sep 11 19:34:18.997475 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.997450 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 19:34:18.998996 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.998976 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 19:34:18.999436 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.999309 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 19:34:18.999726 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:18.999381 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kmcwt\"" Sep 11 19:34:19.001011 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.000948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.003733 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.003715 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 19:34:19.004181 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.003728 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 19:34:19.004303 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.003810 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7lgv6\"" Sep 11 19:34:19.004416 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.003995 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 19:34:19.015017 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.014932 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl"] Sep 11 19:34:19.043434 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46c1d939-adf7-440c-ab3a-91544f68a055-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.043637 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.043637 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.043637 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-root\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.043843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfqx\" (UniqueName: \"kubernetes.io/projected/46c1d939-adf7-440c-ab3a-91544f68a055-kube-api-access-lnfqx\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.043843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b426d294-f15a-4131-a1e9-aac7f1d583ee-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.043843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkklg\" (UniqueName: \"kubernetes.io/projected/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-api-access-bkklg\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.043843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46c1d939-adf7-440c-ab3a-91544f68a055-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.044053 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.043900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.044053 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.044182 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b426d294-f15a-4131-a1e9-aac7f1d583ee-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.044246 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-sys\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.044246 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b61bd66-ef41-47dc-a16e-007810f7d3c3-metrics-client-ca\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.044348 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-textfile\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.044348 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.044348 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46c1d939-adf7-440c-ab3a-91544f68a055-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.044490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-tls\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.044490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-wtmp\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.044490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.044404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtml\" (UniqueName: \"kubernetes.io/projected/1b61bd66-ef41-47dc-a16e-007810f7d3c3-kube-api-access-wbtml\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.059907 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.059875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:19.144979 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.144944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-sys\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.144979 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.144977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b61bd66-ef41-47dc-a16e-007810f7d3c3-metrics-client-ca\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.144997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-textfile\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46c1d939-adf7-440c-ab3a-91544f68a055-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-sys\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-tls\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-wtmp\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtml\" (UniqueName: \"kubernetes.io/projected/1b61bd66-ef41-47dc-a16e-007810f7d3c3-kube-api-access-wbtml\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46c1d939-adf7-440c-ab3a-91544f68a055-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.145285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.145686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-root\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.145686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfqx\" (UniqueName: \"kubernetes.io/projected/46c1d939-adf7-440c-ab3a-91544f68a055-kube-api-access-lnfqx\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.145686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b426d294-f15a-4131-a1e9-aac7f1d583ee-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.145686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkklg\" (UniqueName: \"kubernetes.io/projected/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-api-access-bkklg\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.145686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46c1d939-adf7-440c-ab3a-91544f68a055-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.145686 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.145540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-root\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.146353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.146254 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b426d294-f15a-4131-a1e9-aac7f1d583ee-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.146353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.146320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:19.146514 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:19.146428 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 19:34:19.146514 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.146453 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.146514 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:19.146486 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-tls podName:1b61bd66-ef41-47dc-a16e-007810f7d3c3 nodeName:}" failed. No retries permitted until 2026-09-11 19:34:19.646467161 +0000 UTC m=+292.779779732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-tls") pod "node-exporter-78drv" (UID: "1b61bd66-ef41-47dc-a16e-007810f7d3c3") : secret "node-exporter-tls" not found Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.146714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-wtmp\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.146771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.146801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.146858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b426d294-f15a-4131-a1e9-aac7f1d583ee-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:19.146958 2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:19.147009 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls podName:b426d294-f15a-4131-a1e9-aac7f1d583ee nodeName:}" failed. No retries permitted until 2026-09-11 19:34:19.646991485 +0000 UTC m=+292.780304059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-2f9fl" (UID: "b426d294-f15a-4131-a1e9-aac7f1d583ee") : secret "kube-state-metrics-tls" not found Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.147054 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b61bd66-ef41-47dc-a16e-007810f7d3c3-metrics-client-ca\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.147169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46c1d939-adf7-440c-ab3a-91544f68a055-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.147218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.147181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b426d294-f15a-4131-a1e9-aac7f1d583ee-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.147701 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.147485 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-textfile\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.147750 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.147716 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.148516 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.148475 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46c1d939-adf7-440c-ab3a-91544f68a055-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.149030 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.149008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46c1d939-adf7-440c-ab3a-91544f68a055-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.149881 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.149860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.149974 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.149889 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.158106 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.158071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtml\" (UniqueName: \"kubernetes.io/projected/1b61bd66-ef41-47dc-a16e-007810f7d3c3-kube-api-access-wbtml\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.158247 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.158138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfqx\" (UniqueName: \"kubernetes.io/projected/46c1d939-adf7-440c-ab3a-91544f68a055-kube-api-access-lnfqx\") pod \"openshift-state-metrics-67dcf8c979-z7r2t\" (UID: \"46c1d939-adf7-440c-ab3a-91544f68a055\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.158247 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.158168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkklg\" (UniqueName: \"kubernetes.io/projected/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-api-access-bkklg\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.245575 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.245541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:19.282748 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.282657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" Sep 11 19:34:19.346133 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.346058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:19.512363 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.512331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:19.561684 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.561653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t"] Sep 11 19:34:19.564400 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:34:19.564370 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c1d939_adf7_440c_ab3a_91544f68a055.slice/crio-e8542c08e428956c16270140f1ad42ca5a0fb6bfc2e7ee670da8e300a4057b12 WatchSource:0}: Error finding container e8542c08e428956c16270140f1ad42ca5a0fb6bfc2e7ee670da8e300a4057b12: Status 404 returned error can't find the container with id e8542c08e428956c16270140f1ad42ca5a0fb6bfc2e7ee670da8e300a4057b12 Sep 11 19:34:19.650103 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.650064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:19.650262 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.650138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-tls\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.650322 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:19.650271 2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 11 19:34:19.650365 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:19.650354 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls podName:b426d294-f15a-4131-a1e9-aac7f1d583ee nodeName:}" failed. No retries permitted until 2026-09-11 19:34:20.650333786 +0000 UTC m=+293.783646373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-2f9fl" (UID: "b426d294-f15a-4131-a1e9-aac7f1d583ee") : secret "kube-state-metrics-tls" not found Sep 11 19:34:19.652696 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.652672 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b61bd66-ef41-47dc-a16e-007810f7d3c3-node-exporter-tls\") pod \"node-exporter-78drv\" (UID: \"1b61bd66-ef41-47dc-a16e-007810f7d3c3\") " pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.918334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.918247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-78drv" Sep 11 19:34:19.934522 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:34:19.934488 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b61bd66_ef41_47dc_a16e_007810f7d3c3.slice/crio-0e0d05c4fbb28342717503a0b74371bf4352cb7dc34490e2eb150e6a449b2936 WatchSource:0}: Error finding container 0e0d05c4fbb28342717503a0b74371bf4352cb7dc34490e2eb150e6a449b2936: Status 404 returned error can't find the container with id 0e0d05c4fbb28342717503a0b74371bf4352cb7dc34490e2eb150e6a449b2936 Sep 11 19:34:19.965011 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:19.964975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:20.039001 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.038969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:20.165174 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.165121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:20.363910 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.363866 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" event={"ID":"46c1d939-adf7-440c-ab3a-91544f68a055","Type":"ContainerStarted","Data":"f319927d8b8d118cd50980a0412760e8af0a342e32dc6409639afcee451b6b52"} Sep 11 19:34:20.364102 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.363947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" event={"ID":"46c1d939-adf7-440c-ab3a-91544f68a055","Type":"ContainerStarted","Data":"ac683041ab80432c8c908eee9d59dea2147b58d587be5b848362f76f936c9179"} Sep 11 19:34:20.364102 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.363977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" event={"ID":"46c1d939-adf7-440c-ab3a-91544f68a055","Type":"ContainerStarted","Data":"e8542c08e428956c16270140f1ad42ca5a0fb6bfc2e7ee670da8e300a4057b12"} Sep 11 19:34:20.365334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.365304 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78drv" event={"ID":"1b61bd66-ef41-47dc-a16e-007810f7d3c3","Type":"ContainerStarted","Data":"0e0d05c4fbb28342717503a0b74371bf4352cb7dc34490e2eb150e6a449b2936"} Sep 11 19:34:20.389804 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:20.389758 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-k6shn" podUID="dcd00465-6964-4cd1-8dd5-fda2a726811a" Sep 11 19:34:20.659472 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.659374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:20.663137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.663102 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b426d294-f15a-4131-a1e9-aac7f1d583ee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2f9fl\" (UID: \"b426d294-f15a-4131-a1e9-aac7f1d583ee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:20.782089 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.782020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:20.811002 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:20.810963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" Sep 11 19:34:21.264845 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:21.264745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl"] Sep 11 19:34:21.370642 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:21.370604 2579 generic.go:356] "Generic (PLEG): container finished" podID="1b61bd66-ef41-47dc-a16e-007810f7d3c3" containerID="f0c61b8c3078cc59779ee20c7d96b42495da2095cee1191f55b9d92285bc6824" exitCode=0 Sep 11 19:34:21.370870 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:21.370692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78drv" event={"ID":"1b61bd66-ef41-47dc-a16e-007810f7d3c3","Type":"ContainerDied","Data":"f0c61b8c3078cc59779ee20c7d96b42495da2095cee1191f55b9d92285bc6824"} Sep 11 19:34:21.371980 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:21.371955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" event={"ID":"b426d294-f15a-4131-a1e9-aac7f1d583ee","Type":"ContainerStarted","Data":"6b0894f22109cbc1a3e4540c27586c9e8e30cc64408b9fde8757704e5ff613c7"} Sep 11 19:34:21.374169 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:21.374146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" event={"ID":"46c1d939-adf7-440c-ab3a-91544f68a055","Type":"ContainerStarted","Data":"e01e16c00aaab0b50e9960b739562d390caa51053b6677e3497a315db8052ddf"} Sep 11 19:34:21.405134 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:21.405084 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-z7r2t" podStartSLOduration=2.103581593 podStartE2EDuration="3.405067279s" totalImagesPullingTime="1.301485686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:21.404732921 +0000 UTC m=+294.538045534" watchObservedRunningTime="2026-09-11 19:34:21.405067279 +0000 UTC m=+294.538379880" Sep 11 19:34:21.901640 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:21.901599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:22.001557 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:22.001524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:22.102584 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:22.102535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:22.122624 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:22.122579 2579 patch_prober.go:28] interesting pod/image-registry-7b65645469-d2zfh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 19:34:22.122781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:22.122638 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" podUID="671d3002-744e-4a70-84b7-804ed2df8f86" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 19:34:22.203092 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:22.202997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:22.382765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:22.382708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78drv" event={"ID":"1b61bd66-ef41-47dc-a16e-007810f7d3c3","Type":"ContainerStarted","Data":"6684c33f19662103f0bb69dea1fb289bdcedaae5408bef674ae071dee8d57ce1"} Sep 11 19:34:22.383238 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:22.382797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78drv" event={"ID":"1b61bd66-ef41-47dc-a16e-007810f7d3c3","Type":"ContainerStarted","Data":"04963b81d80d9a064b16c6d79ca4b5e8011e5d5117e96f8f07bf2833dd9d10cc"} Sep 11 19:34:22.404351 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:22.404290 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-78drv" podStartSLOduration=3.27225845 podStartE2EDuration="4.404273207s" totalImagesPullingTime="1.058239112s" totalInitContainerRuntime="73.775645ms" isStatefulPod=true podCreationTimestamp="2026-09-11 19:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:22.402075341 +0000 UTC m=+295.535387930" watchObservedRunningTime="2026-09-11 19:34:22.404273207 +0000 UTC m=+295.537585795" Sep 11 19:34:23.281840 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.281794 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5dbcf4575-fctvc"] Sep 11 19:34:23.284992 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.284971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.289029 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.289006 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 19:34:23.289188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.289038 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 19:34:23.289188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.289092 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-983hqfu5glh0h\"" Sep 11 19:34:23.289188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.289154 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 19:34:23.289381 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.289269 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wvb7k\"" Sep 11 19:34:23.289381 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.289320 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 19:34:23.295187 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.295162 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dbcf4575-fctvc"] Sep 11 19:34:23.337309 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.337275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n7scq" Sep 11 19:34:23.384423 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.384393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-secret-metrics-server-tls\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.384853 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.384465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-secret-metrics-server-client-certs\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.384853 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.384506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-client-ca-bundle\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.384853 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.384591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nkr\" (UniqueName: \"kubernetes.io/projected/323b7006-9597-4d99-8ce5-148dd3318201-kube-api-access-c4nkr\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.384853 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.384631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/323b7006-9597-4d99-8ce5-148dd3318201-metrics-server-audit-profiles\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.384853 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.384656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323b7006-9597-4d99-8ce5-148dd3318201-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.384853 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.384846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/323b7006-9597-4d99-8ce5-148dd3318201-audit-log\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.392431 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.392399 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" event={"ID":"b426d294-f15a-4131-a1e9-aac7f1d583ee","Type":"ContainerStarted","Data":"15ed8b041d5656e0528c70380eabb3ddd98d8e7d99d02935960a2b2199da8c65"} Sep 11 19:34:23.392559 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.392468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" event={"ID":"b426d294-f15a-4131-a1e9-aac7f1d583ee","Type":"ContainerStarted","Data":"52ac1bd3ddfffe3c881915ec9fb91cd70528e233f52c2e394698ff99eb00d080"} Sep 11 19:34:23.392559 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.392500 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" event={"ID":"b426d294-f15a-4131-a1e9-aac7f1d583ee","Type":"ContainerStarted","Data":"f872dc99322c7f5741429f66f70b2ecc7f51f633a1181f3e43e5b3e6e21b0b65"} Sep 11 19:34:23.408069 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.408023 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2f9fl" podStartSLOduration=4.097997662 podStartE2EDuration="5.408006303s" totalImagesPullingTime="1.310008641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:23.406490639 +0000 UTC m=+296.539803224" watchObservedRunningTime="2026-09-11 19:34:23.408006303 +0000 UTC m=+296.541318892" Sep 11 19:34:23.485353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.485316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-secret-metrics-server-tls\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.485549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.485395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-secret-metrics-server-client-certs\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.485549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.485414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-client-ca-bundle\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.485549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.485500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nkr\" (UniqueName: \"kubernetes.io/projected/323b7006-9597-4d99-8ce5-148dd3318201-kube-api-access-c4nkr\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.485549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.485530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/323b7006-9597-4d99-8ce5-148dd3318201-metrics-server-audit-profiles\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.485714 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.485567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323b7006-9597-4d99-8ce5-148dd3318201-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.486377 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.486131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/323b7006-9597-4d99-8ce5-148dd3318201-audit-log\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.486504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.486482 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/323b7006-9597-4d99-8ce5-148dd3318201-audit-log\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.492410 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.488899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-secret-metrics-server-tls\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.492410 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.489147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/323b7006-9597-4d99-8ce5-148dd3318201-metrics-server-audit-profiles\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.492410 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.489965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-client-ca-bundle\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.493243 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.493215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323b7006-9597-4d99-8ce5-148dd3318201-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.494203 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.494180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/323b7006-9597-4d99-8ce5-148dd3318201-secret-metrics-server-client-certs\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.495864 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.495843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nkr\" (UniqueName: \"kubernetes.io/projected/323b7006-9597-4d99-8ce5-148dd3318201-kube-api-access-c4nkr\") pod \"metrics-server-5dbcf4575-fctvc\" (UID: \"323b7006-9597-4d99-8ce5-148dd3318201\") " pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.594522 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.594476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:23.735244 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.735211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:23.871150 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:23.871061 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dbcf4575-fctvc"] Sep 11 19:34:23.874360 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:34:23.874316 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323b7006_9597_4d99_8ce5_148dd3318201.slice/crio-9c4d70db127a315bb9401efe7c78ef5ec886d7d219d90fa2a859cd1410ccdf5d WatchSource:0}: Error finding container 9c4d70db127a315bb9401efe7c78ef5ec886d7d219d90fa2a859cd1410ccdf5d: Status 404 returned error can't find the container with id 9c4d70db127a315bb9401efe7c78ef5ec886d7d219d90fa2a859cd1410ccdf5d Sep 11 19:34:24.128065 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:24.127980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:24.201982 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:24.201951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:24.297786 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:24.297751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:24.394540 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:24.394457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" event={"ID":"323b7006-9597-4d99-8ce5-148dd3318201","Type":"ContainerStarted","Data":"9c4d70db127a315bb9401efe7c78ef5ec886d7d219d90fa2a859cd1410ccdf5d"} Sep 11 19:34:24.398028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:24.398002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:24.454105 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:24.454078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:25.162536 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.162510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:25.169798 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.169763 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:34:25.174788 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.174133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.177842 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.177736 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 19:34:25.177842 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.177803 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 19:34:25.177842 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.177841 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cpqg8\"" Sep 11 19:34:25.178117 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.178025 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 19:34:25.178164 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.178115 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 19:34:25.178164 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.178123 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 19:34:25.178369 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.178341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 19:34:25.178612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.178490 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 19:34:25.179045 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.179023 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 19:34:25.179177 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.179157 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 19:34:25.179875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.179855 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 19:34:25.179998 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.179977 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 19:34:25.180105 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.180050 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 19:34:25.180272 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.180253 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 19:34:25.180655 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.180557 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-gp37ai520bfi\"" Sep 11 19:34:25.181522 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.181500 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 19:34:25.186967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.186913 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:34:25.202760 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.202722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.202983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.202772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.202983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.202809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.202983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.202855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.202983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.202881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.202983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.202912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-config\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203799 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203799 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7tjf\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-kube-api-access-j7tjf\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203799 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.203799 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.203377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304315 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304501 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304501 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304501 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304501 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304501 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-config\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304501 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304501 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7tjf\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-kube-api-access-j7tjf\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.304895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.304713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.305466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.305416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.305466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.305421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.306184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.305992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.306184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.306056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.306391 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.306369 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.307978 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.307660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.308960 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.308627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.309078 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.308985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.309334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.309305 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.310051 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.309794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.310861 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.310450 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.310861 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.310775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.311010 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.310869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.311010 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.310955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.311010 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.310957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.311371 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.311347 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.311788 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.311763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.312166 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.312149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-config\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.313329 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.313303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.320644 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.320613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7tjf\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-kube-api-access-j7tjf\") pod \"prometheus-k8s-0\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.490680 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.490592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:25.795592 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:25.795560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:34:25.799774 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:34:25.799748 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6134677e_4243_42e8_8895_6293fcc85d3c.slice/crio-ae643e93181cdca50003ef7f70e4152115273a9af6b72fb0871fd5a1b43023f3 WatchSource:0}: Error finding container ae643e93181cdca50003ef7f70e4152115273a9af6b72fb0871fd5a1b43023f3: Status 404 returned error can't find the container with id ae643e93181cdca50003ef7f70e4152115273a9af6b72fb0871fd5a1b43023f3 Sep 11 19:34:26.406246 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:26.406208 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerStarted","Data":"ae643e93181cdca50003ef7f70e4152115273a9af6b72fb0871fd5a1b43023f3"} Sep 11 19:34:26.408181 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:26.408150 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" event={"ID":"323b7006-9597-4d99-8ce5-148dd3318201","Type":"ContainerStarted","Data":"ec9327d9a821761cc6687b44a56e9ec186a2fb995bda19d61f1a20c15eb62bfb"} Sep 11 19:34:26.426118 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:26.426061 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" podStartSLOduration=1.790294411 podStartE2EDuration="3.426045112s" totalImagesPullingTime="1.635750701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:26.42465556 +0000 UTC m=+299.557968151" watchObservedRunningTime="2026-09-11 19:34:26.426045112 +0000 UTC m=+299.559357700" Sep 11 19:34:27.134866 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.134782 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" podUID="671d3002-744e-4a70-84b7-804ed2df8f86" containerName="registry" containerID="cri-o://55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23" gracePeriod=30 Sep 11 19:34:27.295975 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.295948 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:34:27.296188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.296104 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:34:27.300688 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.300665 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 19:34:27.395286 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.395263 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:34:27.414990 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.414633 2579 generic.go:356] "Generic (PLEG): container finished" podID="671d3002-744e-4a70-84b7-804ed2df8f86" containerID="55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23" exitCode=0 Sep 11 19:34:27.414990 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.414837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" event={"ID":"671d3002-744e-4a70-84b7-804ed2df8f86","Type":"ContainerDied","Data":"55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23"} Sep 11 19:34:27.414990 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.414896 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" event={"ID":"671d3002-744e-4a70-84b7-804ed2df8f86","Type":"ContainerDied","Data":"d13c0041efb6816f1208b3a7c1de45a40086fe2b5fe444b7fab656980d49b9ed"} Sep 11 19:34:27.414990 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.414922 2579 scope.go:117] "RemoveContainer" containerID="55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23" Sep 11 19:34:27.415426 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.415265 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b65645469-d2zfh" Sep 11 19:34:27.418708 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.418034 2579 generic.go:356] "Generic (PLEG): container finished" podID="6134677e-4243-42e8-8895-6293fcc85d3c" containerID="bb56e8e88c4f2a7e386c15e5b15393af127e407d0a8f176e6c285bab0c2411c0" exitCode=0 Sep 11 19:34:27.418862 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.418699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerDied","Data":"bb56e8e88c4f2a7e386c15e5b15393af127e407d0a8f176e6c285bab0c2411c0"} Sep 11 19:34:27.426236 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.426212 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/671d3002-744e-4a70-84b7-804ed2df8f86-ca-trust-extracted\") pod \"671d3002-744e-4a70-84b7-804ed2df8f86\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " Sep 11 19:34:27.426366 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.426247 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-registry-certificates\") pod \"671d3002-744e-4a70-84b7-804ed2df8f86\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " Sep 11 19:34:27.426366 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.426262 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-trusted-ca\") pod \"671d3002-744e-4a70-84b7-804ed2df8f86\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " Sep 11 19:34:27.426366 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.426295 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-image-registry-private-configuration\") pod \"671d3002-744e-4a70-84b7-804ed2df8f86\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " Sep 11 19:34:27.426528 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.426482 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-bound-sa-token\") pod \"671d3002-744e-4a70-84b7-804ed2df8f86\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " Sep 11 19:34:27.426608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.426591 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") pod \"671d3002-744e-4a70-84b7-804ed2df8f86\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " Sep 11 19:34:27.426888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.426692 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "671d3002-744e-4a70-84b7-804ed2df8f86" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:34:27.426997 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.426967 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "671d3002-744e-4a70-84b7-804ed2df8f86" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:34:27.427141 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.427124 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlb95\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-kube-api-access-qlb95\") pod \"671d3002-744e-4a70-84b7-804ed2df8f86\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " Sep 11 19:34:27.429195 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.429142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "671d3002-744e-4a70-84b7-804ed2df8f86" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:34:27.429390 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.429372 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-installation-pull-secrets\") pod \"671d3002-744e-4a70-84b7-804ed2df8f86\" (UID: \"671d3002-744e-4a70-84b7-804ed2df8f86\") " Sep 11 19:34:27.429680 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.429469 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "671d3002-744e-4a70-84b7-804ed2df8f86" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:34:27.429680 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.429578 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "671d3002-744e-4a70-84b7-804ed2df8f86" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:34:27.430420 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.430183 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-registry-certificates\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:34:27.430624 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.430599 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/671d3002-744e-4a70-84b7-804ed2df8f86-trusted-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:34:27.430719 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.430628 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-image-registry-private-configuration\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:34:27.430719 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.430633 2579 scope.go:117] "RemoveContainer" containerID="55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23" Sep 11 19:34:27.430719 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.430644 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-bound-sa-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:34:27.430719 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.430657 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-registry-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:34:27.431268 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:34:27.431111 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23\": container with ID starting with 55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23 not found: ID does not exist" containerID="55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23" Sep 11 19:34:27.431268 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.431145 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23"} err="failed to get container status \"55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23\": rpc error: code = NotFound desc = could not find container \"55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23\": container with ID starting with 55136bd2fc98ea19e2bab307921819b001e0e7967e3680d70c9cec355ee37d23 not found: ID does not exist" Sep 11 19:34:27.431384 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.431321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-kube-api-access-qlb95" (OuterVolumeSpecName: "kube-api-access-qlb95") pod "671d3002-744e-4a70-84b7-804ed2df8f86" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86"). InnerVolumeSpecName "kube-api-access-qlb95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:34:27.431858 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.431812 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "671d3002-744e-4a70-84b7-804ed2df8f86" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:34:27.436293 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.436266 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671d3002-744e-4a70-84b7-804ed2df8f86-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "671d3002-744e-4a70-84b7-804ed2df8f86" (UID: "671d3002-744e-4a70-84b7-804ed2df8f86"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:34:27.532334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.532257 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlb95\" (UniqueName: \"kubernetes.io/projected/671d3002-744e-4a70-84b7-804ed2df8f86-kube-api-access-qlb95\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:34:27.532334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.532283 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/671d3002-744e-4a70-84b7-804ed2df8f86-installation-pull-secrets\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:34:27.532334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.532296 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/671d3002-744e-4a70-84b7-804ed2df8f86-ca-trust-extracted\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:34:27.736629 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.736596 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b65645469-d2zfh"] Sep 11 19:34:27.740323 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:27.740290 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b65645469-d2zfh"] Sep 11 19:34:29.392705 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:29.392666 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="671d3002-744e-4a70-84b7-804ed2df8f86" path="/var/lib/kubelet/pods/671d3002-744e-4a70-84b7-804ed2df8f86/volumes" Sep 11 19:34:29.956748 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:29.956709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:30.496294 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:30.496248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:30.622287 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:30.622248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:30.782338 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:30.782309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:30.923190 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:30.923172 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:34:30.960773 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:30.960746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:31.390107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:31.390059 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:31.392554 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:31.392538 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:34:31.441844 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:31.441782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerStarted","Data":"9bca9dd13b6001f07438a651f2268c887f86319f72f57217ab534a4a916eba08"} Sep 11 19:34:31.442016 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:31.441878 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerStarted","Data":"a0b58b9792d99c4fd661be766679694ea98dc60c3235e83533609f2e972d4354"} Sep 11 19:34:31.449608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:31.449574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:31.568272 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:31.568208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:31.634170 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:31.634133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:32.175481 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:32.175452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:32.386498 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:32.386449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:32.776731 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:32.776684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:33.457203 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:33.457166 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerStarted","Data":"b3930677698b0654e4cbe779ea4c3d82333657e47485e67a08452bf1130d5114"} Sep 11 19:34:33.457419 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:33.457229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerStarted","Data":"595fd241acbeb0baa93ec566a14c841f596a0a95fe6412cb060763c27ddb0b89"} Sep 11 19:34:33.457419 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:33.457255 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerStarted","Data":"c76a5ede8c780d77455fd96cd5127ab8b9cb138ef0ad37d3f9156cb4144370cf"} Sep 11 19:34:33.457419 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:33.457280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerStarted","Data":"ba70cdb9a7f8cf277c4df847490fe5e6f64c4ecd6880a516eab22e1b84ac1a49"} Sep 11 19:34:33.486272 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:33.486216 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.01158612 podStartE2EDuration="8.486198352s" totalImagesPullingTime="6.457076283s" totalInitContainerRuntime="17.535949ms" isStatefulPod=false podCreationTimestamp="2026-09-11 19:34:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:33.48379137 +0000 UTC m=+306.617103995" watchObservedRunningTime="2026-09-11 19:34:33.486198352 +0000 UTC m=+306.619510960" Sep 11 19:34:33.578469 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:33.578424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:33.638151 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:33.638117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:35.491451 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:35.491417 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:34:37.508668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:37.508634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:37.609400 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:37.609363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:38.346390 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:38.346358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:38.991579 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:38.991549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:39.051217 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:39.051179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:41.796146 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:41.796111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:43.594983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:43.594946 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:43.595451 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:43.595014 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:34:44.601436 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:44.601398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:44.665525 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:44.665493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:46.586368 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:46.586333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:46.687218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:46.687186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:47.582323 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:47.582281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:47.682987 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:47.682949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:50.483728 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:50.483699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:50.584999 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:50.584969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:51.797430 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:51.797400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:51.852308 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:51.852267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:54.001959 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:54.001914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:54.054473 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:54.054442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:56.614227 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:56.614189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:56.670180 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:56.670142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:57.170000 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:57.169966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:57.230813 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:57.230778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:57.810228 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:57.810195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:34:57.862069 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:34:57.862033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:00.565174 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:00.565142 2579 generic.go:356] "Generic (PLEG): container finished" podID="416bd977-648b-46f7-9f45-db0b63a02ac8" containerID="aba1f97708ebdd797738c7465b41a24dc2ee78641f3539918420e2cfe6a550c4" exitCode=0 Sep 11 19:35:00.565546 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:00.565183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6g9vp" event={"ID":"416bd977-648b-46f7-9f45-db0b63a02ac8","Type":"ContainerDied","Data":"aba1f97708ebdd797738c7465b41a24dc2ee78641f3539918420e2cfe6a550c4"} Sep 11 19:35:00.565585 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:00.565544 2579 scope.go:117] "RemoveContainer" containerID="aba1f97708ebdd797738c7465b41a24dc2ee78641f3539918420e2cfe6a550c4" Sep 11 19:35:01.235089 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:01.235062 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k5nzx_939f0c9d-6cab-4c64-adcf-8f3052559d25/serve-healthcheck-canary/0.log" Sep 11 19:35:01.571364 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:01.571331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6g9vp" event={"ID":"416bd977-648b-46f7-9f45-db0b63a02ac8","Type":"ContainerStarted","Data":"569230940b688b79da0f08976daf20f5c694d0fa2279f60b5a5d8f8352e098ed"} Sep 11 19:35:03.600102 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:03.600069 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:35:03.604241 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:03.604216 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5dbcf4575-fctvc" Sep 11 19:35:09.882967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:09.882937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:10.053613 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:10.053580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:10.151577 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:10.151438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:10.607644 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:10.607612 2579 generic.go:356] "Generic (PLEG): container finished" podID="f7996815-d98c-4995-bb2d-25f561e3ce4b" containerID="bd01ed4e6a7d650441ff49112c7096d63d4b6ffe3b78bd20618be644a5c91628" exitCode=0 Sep 11 19:35:10.607860 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:10.607686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" event={"ID":"f7996815-d98c-4995-bb2d-25f561e3ce4b","Type":"ContainerDied","Data":"bd01ed4e6a7d650441ff49112c7096d63d4b6ffe3b78bd20618be644a5c91628"} Sep 11 19:35:10.608069 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:10.608054 2579 scope.go:117] "RemoveContainer" containerID="bd01ed4e6a7d650441ff49112c7096d63d4b6ffe3b78bd20618be644a5c91628" Sep 11 19:35:11.614120 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:11.614086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z6c62" event={"ID":"f7996815-d98c-4995-bb2d-25f561e3ce4b","Type":"ContainerStarted","Data":"0c6ec71721b2d31f066422d3d1aaed59c0a3cb80be1e20a2fefbc3321b16f6a0"} Sep 11 19:35:20.490475 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:20.490373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:20.589276 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:20.589241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:25.491411 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:25.491365 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:25.512195 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:25.512164 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:25.690325 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:25.690294 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:38.140093 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:38.140057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:38.441218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:38.441132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:40.321049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:40.320999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:40.437066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:40.436998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:41.289037 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:41.289000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:35:41.291540 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:41.291521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd00465-6964-4cd1-8dd5-fda2a726811a-metrics-certs\") pod \"network-metrics-daemon-k6shn\" (UID: \"dcd00465-6964-4cd1-8dd5-fda2a726811a\") " pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:35:41.296624 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:41.296603 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7w7kq\"" Sep 11 19:35:41.304346 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:41.304310 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6shn" Sep 11 19:35:41.582372 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:41.582283 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k6shn"] Sep 11 19:35:41.587011 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:35:41.586981 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd00465_6964_4cd1_8dd5_fda2a726811a.slice/crio-a5827734f0cb0c0785012413827d7c2cb6e1a27c4c1b6ac4897d460602c6cbff WatchSource:0}: Error finding container a5827734f0cb0c0785012413827d7c2cb6e1a27c4c1b6ac4897d460602c6cbff: Status 404 returned error can't find the container with id a5827734f0cb0c0785012413827d7c2cb6e1a27c4c1b6ac4897d460602c6cbff Sep 11 19:35:41.736802 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:41.736764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6shn" event={"ID":"dcd00465-6964-4cd1-8dd5-fda2a726811a","Type":"ContainerStarted","Data":"a5827734f0cb0c0785012413827d7c2cb6e1a27c4c1b6ac4897d460602c6cbff"} Sep 11 19:35:42.742562 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:42.742530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6shn" event={"ID":"dcd00465-6964-4cd1-8dd5-fda2a726811a","Type":"ContainerStarted","Data":"a16954dbf27ba431b78cf9b53c8db85bcc014d02c100ed16a51ea7c95c382977"} Sep 11 19:35:42.742981 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:42.742584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6shn" event={"ID":"dcd00465-6964-4cd1-8dd5-fda2a726811a","Type":"ContainerStarted","Data":"f8bb5aa387596ceaf16f4cab37ba32fbcdaa16756bcdea6bcd5a4d9e6ff3a76a"} Sep 11 19:35:42.748168 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:42.748139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:42.763476 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:42.763378 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k6shn" podStartSLOduration=374.846771109 podStartE2EDuration="6m15.763359688s" totalImagesPullingTime="916.588579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:42.761973473 +0000 UTC m=+375.895286058" watchObservedRunningTime="2026-09-11 19:35:42.763359688 +0000 UTC m=+375.896672276" Sep 11 19:35:42.848545 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:42.848508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:42.949224 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:42.949193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:43.058662 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.058628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:43.120134 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.120094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:43.549557 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.549521 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:35:43.549842 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.549798 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="prometheus" containerID="cri-o://a0b58b9792d99c4fd661be766679694ea98dc60c3235e83533609f2e972d4354" gracePeriod=600 Sep 11 19:35:43.549967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.549813 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy" containerID="cri-o://595fd241acbeb0baa93ec566a14c841f596a0a95fe6412cb060763c27ddb0b89" gracePeriod=600 Sep 11 19:35:43.549967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.549868 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="thanos-sidecar" containerID="cri-o://ba70cdb9a7f8cf277c4df847490fe5e6f64c4ecd6880a516eab22e1b84ac1a49" gracePeriod=600 Sep 11 19:35:43.549967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.549882 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy-web" containerID="cri-o://c76a5ede8c780d77455fd96cd5127ab8b9cb138ef0ad37d3f9156cb4144370cf" gracePeriod=600 Sep 11 19:35:43.549967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.549925 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b3930677698b0654e4cbe779ea4c3d82333657e47485e67a08452bf1130d5114" gracePeriod=600 Sep 11 19:35:43.550176 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.549989 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="config-reloader" containerID="cri-o://9bca9dd13b6001f07438a651f2268c887f86319f72f57217ab534a4a916eba08" gracePeriod=600 Sep 11 19:35:43.759683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759643 2579 generic.go:356] "Generic (PLEG): container finished" podID="6134677e-4243-42e8-8895-6293fcc85d3c" containerID="b3930677698b0654e4cbe779ea4c3d82333657e47485e67a08452bf1130d5114" exitCode=0 Sep 11 19:35:43.759683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759680 2579 generic.go:356] "Generic (PLEG): container finished" podID="6134677e-4243-42e8-8895-6293fcc85d3c" containerID="595fd241acbeb0baa93ec566a14c841f596a0a95fe6412cb060763c27ddb0b89" exitCode=0 Sep 11 19:35:43.759683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759690 2579 generic.go:356] "Generic (PLEG): container finished" podID="6134677e-4243-42e8-8895-6293fcc85d3c" containerID="c76a5ede8c780d77455fd96cd5127ab8b9cb138ef0ad37d3f9156cb4144370cf" exitCode=0 Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759701 2579 generic.go:356] "Generic (PLEG): container finished" podID="6134677e-4243-42e8-8895-6293fcc85d3c" containerID="ba70cdb9a7f8cf277c4df847490fe5e6f64c4ecd6880a516eab22e1b84ac1a49" exitCode=0 Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759709 2579 generic.go:356] "Generic (PLEG): container finished" podID="6134677e-4243-42e8-8895-6293fcc85d3c" containerID="9bca9dd13b6001f07438a651f2268c887f86319f72f57217ab534a4a916eba08" exitCode=0 Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759717 2579 generic.go:356] "Generic (PLEG): container finished" podID="6134677e-4243-42e8-8895-6293fcc85d3c" containerID="a0b58b9792d99c4fd661be766679694ea98dc60c3235e83533609f2e972d4354" exitCode=0 Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759714 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerDied","Data":"b3930677698b0654e4cbe779ea4c3d82333657e47485e67a08452bf1130d5114"} Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerDied","Data":"595fd241acbeb0baa93ec566a14c841f596a0a95fe6412cb060763c27ddb0b89"} Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerDied","Data":"c76a5ede8c780d77455fd96cd5127ab8b9cb138ef0ad37d3f9156cb4144370cf"} Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759887 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerDied","Data":"ba70cdb9a7f8cf277c4df847490fe5e6f64c4ecd6880a516eab22e1b84ac1a49"} Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759912 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerDied","Data":"9bca9dd13b6001f07438a651f2268c887f86319f72f57217ab534a4a916eba08"} Sep 11 19:35:43.760261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.759940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerDied","Data":"a0b58b9792d99c4fd661be766679694ea98dc60c3235e83533609f2e972d4354"} Sep 11 19:35:43.800954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.800870 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:43.811337 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811315 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-2\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.811481 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811342 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-metrics-client-ca\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.811481 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-config-out\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.811481 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811390 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-web-config\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.811481 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811404 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-config\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.811481 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811423 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7tjf\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-kube-api-access-j7tjf\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.811801 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811696 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:35:43.811958 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811847 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:35:43.811958 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811885 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-1\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.811958 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.811930 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-0\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812127 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-tls-assets\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812127 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812074 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:35:43.812127 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812084 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-tls\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812282 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812282 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812190 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-kube-rbac-proxy\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812282 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812232 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-thanos-prometheus-http-client-file\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812282 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812266 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-trusted-ca-bundle\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812306 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812339 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-serving-certs-ca-bundle\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812376 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-db\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812411 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-grpc-tls\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812441 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-kubelet-serving-ca-bundle\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812466 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-metrics-client-certs\") pod \"6134677e-4243-42e8-8895-6293fcc85d3c\" (UID: \"6134677e-4243-42e8-8895-6293fcc85d3c\") " Sep 11 19:35:43.812779 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812759 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.812863 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812786 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.812863 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.812801 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-metrics-client-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.813511 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.813028 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:35:43.815735 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.814610 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:35:43.815735 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.814791 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:35:43.815735 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.815220 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:35:43.815735 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.815275 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-config-out" (OuterVolumeSpecName: "config-out") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:35:43.815735 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.815700 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:35:43.818723 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.818688 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.819126 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.819092 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:35:43.819126 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.819098 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-kube-api-access-j7tjf" (OuterVolumeSpecName: "kube-api-access-j7tjf") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "kube-api-access-j7tjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:35:43.819287 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.819153 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.819287 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.819196 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-config" (OuterVolumeSpecName: "config") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.819287 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.819257 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.819793 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.819769 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.819917 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.819898 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.820065 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.820040 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.820951 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.820922 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.829782 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.829757 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-web-config" (OuterVolumeSpecName: "web-config") pod "6134677e-4243-42e8-8895-6293fcc85d3c" (UID: "6134677e-4243-42e8-8895-6293fcc85d3c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:35:43.914154 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914119 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914154 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914150 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914164 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-db\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914178 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-grpc-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914190 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914204 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-metrics-client-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914214 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6134677e-4243-42e8-8895-6293fcc85d3c-config-out\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914226 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-web-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914236 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914246 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7tjf\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-kube-api-access-j7tjf\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914256 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914267 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6134677e-4243-42e8-8895-6293fcc85d3c-tls-assets\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914278 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914290 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914303 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-secret-kube-rbac-proxy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914315 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6134677e-4243-42e8-8895-6293fcc85d3c-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:43.914392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:43.914327 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6134677e-4243-42e8-8895-6293fcc85d3c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:35:44.767494 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.767461 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6134677e-4243-42e8-8895-6293fcc85d3c","Type":"ContainerDied","Data":"ae643e93181cdca50003ef7f70e4152115273a9af6b72fb0871fd5a1b43023f3"} Sep 11 19:35:44.767866 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.767523 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.767866 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.767535 2579 scope.go:117] "RemoveContainer" containerID="b3930677698b0654e4cbe779ea4c3d82333657e47485e67a08452bf1130d5114" Sep 11 19:35:44.775949 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.775932 2579 scope.go:117] "RemoveContainer" containerID="595fd241acbeb0baa93ec566a14c841f596a0a95fe6412cb060763c27ddb0b89" Sep 11 19:35:44.784335 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.784313 2579 scope.go:117] "RemoveContainer" containerID="c76a5ede8c780d77455fd96cd5127ab8b9cb138ef0ad37d3f9156cb4144370cf" Sep 11 19:35:44.795504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.795460 2579 scope.go:117] "RemoveContainer" containerID="ba70cdb9a7f8cf277c4df847490fe5e6f64c4ecd6880a516eab22e1b84ac1a49" Sep 11 19:35:44.796296 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.796274 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:35:44.803025 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.802995 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:35:44.804143 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.804125 2579 scope.go:117] "RemoveContainer" containerID="9bca9dd13b6001f07438a651f2268c887f86319f72f57217ab534a4a916eba08" Sep 11 19:35:44.812003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.811981 2579 scope.go:117] "RemoveContainer" containerID="a0b58b9792d99c4fd661be766679694ea98dc60c3235e83533609f2e972d4354" Sep 11 19:35:44.819900 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.819873 2579 scope.go:117] "RemoveContainer" containerID="bb56e8e88c4f2a7e386c15e5b15393af127e407d0a8f176e6c285bab0c2411c0" Sep 11 19:35:44.846308 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846273 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:35:44.846570 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846558 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="prometheus" Sep 11 19:35:44.846612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846572 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="prometheus" Sep 11 19:35:44.846612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846580 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy-web" Sep 11 19:35:44.846612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846585 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy-web" Sep 11 19:35:44.846612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846595 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="config-reloader" Sep 11 19:35:44.846612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846602 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="config-reloader" Sep 11 19:35:44.846612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846611 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="thanos-sidecar" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846617 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="thanos-sidecar" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846624 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy-thanos" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846629 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy-thanos" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846636 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846641 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846647 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671d3002-744e-4a70-84b7-804ed2df8f86" containerName="registry" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846652 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="671d3002-744e-4a70-84b7-804ed2df8f86" containerName="registry" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846658 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="init-config-reloader" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846663 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="init-config-reloader" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846714 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy-web" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846722 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="prometheus" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846729 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="thanos-sidecar" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846735 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="671d3002-744e-4a70-84b7-804ed2df8f86" containerName="registry" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846740 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="config-reloader" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846745 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy" Sep 11 19:35:44.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.846749 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" containerName="kube-rbac-proxy-thanos" Sep 11 19:35:44.852306 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.852284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.854932 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.854912 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 19:35:44.856284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.856271 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 19:35:44.856763 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.856750 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 19:35:44.857020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.856989 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cpqg8\"" Sep 11 19:35:44.857020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857007 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 19:35:44.857020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857013 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 19:35:44.857278 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857261 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 19:35:44.857337 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857277 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 19:35:44.857520 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857506 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-gp37ai520bfi\"" Sep 11 19:35:44.857599 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857585 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 19:35:44.857775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857761 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 19:35:44.857852 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857783 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 19:35:44.857922 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.857899 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 19:35:44.858542 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.858524 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 19:35:44.863362 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.863343 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 19:35:44.864894 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.864874 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 19:35:44.867881 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.867862 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:35:44.923396 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-config\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923396 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923762 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxl8\" (UniqueName: \"kubernetes.io/projected/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-kube-api-access-tsxl8\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923762 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923762 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923762 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923762 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:44.923985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:44.923969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025230 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025230 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-config\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025230 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025230 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxl8\" (UniqueName: \"kubernetes.io/projected/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-kube-api-access-tsxl8\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.025687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.026116 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.025925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.026232 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.026204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.028567 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.028469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.028567 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.028469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.028753 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.028611 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-config\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.029466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.028921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.029466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.028979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.029466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.028988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.029466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.029022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.029466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.029251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.029466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.029408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.030079 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.029476 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.030079 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.029813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.030079 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.030058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.030767 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.030741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.031558 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.031531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.031796 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.031777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.031974 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.031958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.032082 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.032066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.034905 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.034883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxl8\" (UniqueName: \"kubernetes.io/projected/ddcaf05d-a1e5-4275-b2a5-d231fdabca9f-kube-api-access-tsxl8\") pod \"prometheus-k8s-0\" (UID: \"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.161772 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.161675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:45.392303 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.392262 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6134677e-4243-42e8-8895-6293fcc85d3c" path="/var/lib/kubelet/pods/6134677e-4243-42e8-8895-6293fcc85d3c/volumes" Sep 11 19:35:45.456207 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.456181 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 19:35:45.456644 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:35:45.456618 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcaf05d_a1e5_4275_b2a5_d231fdabca9f.slice/crio-128462c9d96f0e01fb20246042eef0c07ab1a67a0b84fe95b14c8df28d763bba WatchSource:0}: Error finding container 128462c9d96f0e01fb20246042eef0c07ab1a67a0b84fe95b14c8df28d763bba: Status 404 returned error can't find the container with id 128462c9d96f0e01fb20246042eef0c07ab1a67a0b84fe95b14c8df28d763bba Sep 11 19:35:45.772812 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.772723 2579 generic.go:356] "Generic (PLEG): container finished" podID="ddcaf05d-a1e5-4275-b2a5-d231fdabca9f" containerID="bb3e63d19bd284a25149f00c5083f60c7eeb92a1e60b420b28dfb70e293e4c5a" exitCode=0 Sep 11 19:35:45.773263 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.772814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f","Type":"ContainerDied","Data":"bb3e63d19bd284a25149f00c5083f60c7eeb92a1e60b420b28dfb70e293e4c5a"} Sep 11 19:35:45.773263 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:45.772910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f","Type":"ContainerStarted","Data":"128462c9d96f0e01fb20246042eef0c07ab1a67a0b84fe95b14c8df28d763bba"} Sep 11 19:35:46.766903 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.766868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:46.781991 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.781956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f","Type":"ContainerStarted","Data":"0897287a021fb842d26eedf10842ea2b802702ef22f3080a44d3484eebf920e0"} Sep 11 19:35:46.782390 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.782055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f","Type":"ContainerStarted","Data":"2c40019b1977cdf0b8a7b1548642fde96ffa1d5512d7844be21e783280b16d4f"} Sep 11 19:35:46.782390 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.782088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f","Type":"ContainerStarted","Data":"32d71f6ad59eb8cd18bd4ad990fa551125bd2a6db6011adee9af6286cab3ef04"} Sep 11 19:35:46.782390 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.782114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f","Type":"ContainerStarted","Data":"c38e4491bb132a6523c7be5f72609c7ec79fe1c3c84f588a3eac9c4d757a07ad"} Sep 11 19:35:46.782390 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.782136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f","Type":"ContainerStarted","Data":"60da6077a7a31d03867cce3ce563c4d2aee9037cb4d1f660bec02d2541a9fa41"} Sep 11 19:35:46.782390 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.782158 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddcaf05d-a1e5-4275-b2a5-d231fdabca9f","Type":"ContainerStarted","Data":"35f4561f4c95fcd98680ddd42c3f0cf6600e923bba412542369536c3758eadd5"} Sep 11 19:35:46.817209 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.817150 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.798034683 podStartE2EDuration="2.817131564s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.096881ms" isStatefulPod=false podCreationTimestamp="2026-09-11 19:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:46.81509865 +0000 UTC m=+379.948411265" watchObservedRunningTime="2026-09-11 19:35:46.817131564 +0000 UTC m=+379.950444153" Sep 11 19:35:46.831128 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:46.831099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:50.162789 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:50.162751 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:35:57.802886 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:57.802850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:57.866574 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:57.866541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:35:57.927898 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:35:57.927849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:36:07.027087 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:07.027046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:36:07.081537 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:07.081502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:36:31.313579 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:31.313531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:36:31.411250 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:31.411210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:36:45.162464 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:45.162425 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:36:45.179060 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:45.179035 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:36:46.045742 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:46.045704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 19:36:53.428467 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:53.428349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:36:53.481721 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:36:53.481671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:04.485712 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:04.485676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:04.553481 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:04.553449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:05.393586 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:05.393550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:05.450032 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:05.449993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:15.114034 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:15.113996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:15.166538 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:15.166501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:15.221727 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:15.221676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:20.496168 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:20.496130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:20.596452 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:20.596419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:37:20.768147 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:37:20.768028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:04.198758 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:04.198723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:04.336323 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:04.336288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:15.265486 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:15.265452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:26.558715 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:26.558641 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:27.944901 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:27.944848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:30.175658 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:30.175613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:30.276434 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:30.276400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:35.153706 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:35.153653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:35.253458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:35.253430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:45.138083 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:45.138048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:45.229068 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:45.228964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:45.383100 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:45.383064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:45.484802 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:45.484704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:47.761146 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:47.761111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:49.100298 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:49.100270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:51.931846 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:51.931793 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-98hq8"] Sep 11 19:38:51.935034 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:51.935016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:51.938925 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:51.938903 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 19:38:51.939041 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:51.938949 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 19:38:51.940143 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:51.940121 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-v4r2h\"" Sep 11 19:38:51.968186 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:51.968157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-98hq8"] Sep 11 19:38:51.989209 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:51.989182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46c3ee5b-f347-4f9c-aafd-d29d7b566539-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-98hq8\" (UID: \"46c3ee5b-f347-4f9c-aafd-d29d7b566539\") " pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:51.989391 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:51.989223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62h7\" (UniqueName: \"kubernetes.io/projected/46c3ee5b-f347-4f9c-aafd-d29d7b566539-kube-api-access-x62h7\") pod \"cert-manager-webhook-6486ffd997-98hq8\" (UID: \"46c3ee5b-f347-4f9c-aafd-d29d7b566539\") " pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:52.090389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:52.090301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46c3ee5b-f347-4f9c-aafd-d29d7b566539-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-98hq8\" (UID: \"46c3ee5b-f347-4f9c-aafd-d29d7b566539\") " pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:52.090543 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:52.090433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62h7\" (UniqueName: \"kubernetes.io/projected/46c3ee5b-f347-4f9c-aafd-d29d7b566539-kube-api-access-x62h7\") pod \"cert-manager-webhook-6486ffd997-98hq8\" (UID: \"46c3ee5b-f347-4f9c-aafd-d29d7b566539\") " pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:52.098680 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:52.098650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46c3ee5b-f347-4f9c-aafd-d29d7b566539-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-98hq8\" (UID: \"46c3ee5b-f347-4f9c-aafd-d29d7b566539\") " pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:52.099193 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:52.099175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62h7\" (UniqueName: \"kubernetes.io/projected/46c3ee5b-f347-4f9c-aafd-d29d7b566539-kube-api-access-x62h7\") pod \"cert-manager-webhook-6486ffd997-98hq8\" (UID: \"46c3ee5b-f347-4f9c-aafd-d29d7b566539\") " pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:52.254657 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:52.254569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:52.526030 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:52.526007 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-98hq8"] Sep 11 19:38:52.528378 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:38:52.528353 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46c3ee5b_f347_4f9c_aafd_d29d7b566539.slice/crio-b2546116ee9e7760b6284c61baa243737cc2db754f59a5e9cf89738cf7819c61 WatchSource:0}: Error finding container b2546116ee9e7760b6284c61baa243737cc2db754f59a5e9cf89738cf7819c61: Status 404 returned error can't find the container with id b2546116ee9e7760b6284c61baa243737cc2db754f59a5e9cf89738cf7819c61 Sep 11 19:38:53.533111 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:53.533071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" event={"ID":"46c3ee5b-f347-4f9c-aafd-d29d7b566539","Type":"ContainerStarted","Data":"b2546116ee9e7760b6284c61baa243737cc2db754f59a5e9cf89738cf7819c61"} Sep 11 19:38:53.777957 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:53.777927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:53.870698 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:53.870655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:55.364036 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:55.364005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:55.543322 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:55.543286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" event={"ID":"46c3ee5b-f347-4f9c-aafd-d29d7b566539","Type":"ContainerStarted","Data":"fb7ca92840b5e6327d476ab9f1450977a6a942680577b92ea2b6281c4dcd5192"} Sep 11 19:38:55.543519 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:55.543353 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:38:55.559676 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:55.559621 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" podStartSLOduration=1.6949774739999999 podStartE2EDuration="4.559603312s" totalImagesPullingTime="2.864625838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:55.558614677 +0000 UTC m=+568.691927266" watchObservedRunningTime="2026-09-11 19:38:55.559603312 +0000 UTC m=+568.692915902" Sep 11 19:38:55.650312 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:55.650221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:55.750747 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:55.750690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:56.808020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:56.807987 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k9r6b"] Sep 11 19:38:56.811296 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:56.811274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k9r6b" Sep 11 19:38:56.814302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:56.814278 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2c25f\"" Sep 11 19:38:56.820340 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:56.820313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k9r6b"] Sep 11 19:38:56.935527 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:56.935496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5828\" (UniqueName: \"kubernetes.io/projected/b35cc8d7-6b81-4396-a472-5f0bac11af42-kube-api-access-b5828\") pod \"cert-manager-55b4d954d5-k9r6b\" (UID: \"b35cc8d7-6b81-4396-a472-5f0bac11af42\") " pod="cert-manager/cert-manager-55b4d954d5-k9r6b" Sep 11 19:38:56.935689 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:56.935547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b35cc8d7-6b81-4396-a472-5f0bac11af42-bound-sa-token\") pod \"cert-manager-55b4d954d5-k9r6b\" (UID: \"b35cc8d7-6b81-4396-a472-5f0bac11af42\") " pod="cert-manager/cert-manager-55b4d954d5-k9r6b" Sep 11 19:38:57.036173 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:57.036130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b35cc8d7-6b81-4396-a472-5f0bac11af42-bound-sa-token\") pod \"cert-manager-55b4d954d5-k9r6b\" (UID: \"b35cc8d7-6b81-4396-a472-5f0bac11af42\") " pod="cert-manager/cert-manager-55b4d954d5-k9r6b" Sep 11 19:38:57.036348 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:57.036243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5828\" (UniqueName: \"kubernetes.io/projected/b35cc8d7-6b81-4396-a472-5f0bac11af42-kube-api-access-b5828\") pod \"cert-manager-55b4d954d5-k9r6b\" (UID: \"b35cc8d7-6b81-4396-a472-5f0bac11af42\") " pod="cert-manager/cert-manager-55b4d954d5-k9r6b" Sep 11 19:38:57.045360 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:57.045330 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5828\" (UniqueName: \"kubernetes.io/projected/b35cc8d7-6b81-4396-a472-5f0bac11af42-kube-api-access-b5828\") pod \"cert-manager-55b4d954d5-k9r6b\" (UID: \"b35cc8d7-6b81-4396-a472-5f0bac11af42\") " pod="cert-manager/cert-manager-55b4d954d5-k9r6b" Sep 11 19:38:57.045360 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:57.045350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b35cc8d7-6b81-4396-a472-5f0bac11af42-bound-sa-token\") pod \"cert-manager-55b4d954d5-k9r6b\" (UID: \"b35cc8d7-6b81-4396-a472-5f0bac11af42\") " pod="cert-manager/cert-manager-55b4d954d5-k9r6b" Sep 11 19:38:57.120677 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:57.120593 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k9r6b" Sep 11 19:38:57.393492 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:57.393470 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k9r6b"] Sep 11 19:38:57.395096 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:38:57.395073 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb35cc8d7_6b81_4396_a472_5f0bac11af42.slice/crio-7da0768ed5ac021e1aec0a74cc604d9482d59d50d81143f43172de281a913203 WatchSource:0}: Error finding container 7da0768ed5ac021e1aec0a74cc604d9482d59d50d81143f43172de281a913203: Status 404 returned error can't find the container with id 7da0768ed5ac021e1aec0a74cc604d9482d59d50d81143f43172de281a913203 Sep 11 19:38:57.553565 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:57.553526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k9r6b" event={"ID":"b35cc8d7-6b81-4396-a472-5f0bac11af42","Type":"ContainerStarted","Data":"607e8906769034d07ee4949fea376f611b1eb4d59a3087fbb6fc052f47bcd0a0"} Sep 11 19:38:57.553730 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:57.553575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k9r6b" event={"ID":"b35cc8d7-6b81-4396-a472-5f0bac11af42","Type":"ContainerStarted","Data":"7da0768ed5ac021e1aec0a74cc604d9482d59d50d81143f43172de281a913203"} Sep 11 19:38:58.544964 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:58.544930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:38:58.649209 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:38:58.649180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:00.919778 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:00.919734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:01.549902 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:01.549870 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-98hq8" Sep 11 19:39:01.569928 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:01.569865 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-k9r6b" podStartSLOduration=5.569842395 podStartE2EDuration="5.569842395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:57.569167684 +0000 UTC m=+570.702480273" watchObservedRunningTime="2026-09-11 19:39:01.569842395 +0000 UTC m=+574.703154977" Sep 11 19:39:06.533503 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:06.533450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:18.278648 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:18.278610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:18.376499 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:18.376459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:18.901663 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:18.901623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:19.156782 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.156700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:19.260183 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.260154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:19.317212 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.317173 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm"] Sep 11 19:39:19.322569 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.322548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.327402 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.327379 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 19:39:19.327684 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.327667 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 19:39:19.327952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.327935 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 19:39:19.327952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.327950 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 19:39:19.328082 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.327953 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b7k7l\"" Sep 11 19:39:19.436504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.436407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.436504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.436453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-tmp\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.436504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.436476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7968k\" (UniqueName: \"kubernetes.io/projected/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-kube-api-access-7968k\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.436776 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.436550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-manifests\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.436776 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.436652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.498705 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.498674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:19.537952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.537908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-manifests\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.538197 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.538164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.538367 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.538343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-manifests\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.538444 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.538414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.538554 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.538531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-tmp\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.538620 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.538605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7968k\" (UniqueName: \"kubernetes.io/projected/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-kube-api-access-7968k\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.541080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.541048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-tmp\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.541080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.541072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.541321 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.541302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.547898 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.547873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7968k\" (UniqueName: \"kubernetes.io/projected/3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4-kube-api-access-7968k\") pod \"opendatahub-operator-controller-manager-cccd5967-755nm\" (UID: \"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.599207 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.599172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:19.633333 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.633296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:19.703946 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.703774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:19.803380 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.803343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:19.907044 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:19.907010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:27.324229 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:27.324091 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:39:27.335123 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:27.325475 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:39:39.377260 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:39.377223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:39.477916 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:39.477884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:40.541226 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:40.541193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:40.597188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:40.597149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:44.491795 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:44.491762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:44.602113 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:44.602085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:49.415210 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:49.415126 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm"] Sep 11 19:39:49.704785 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:49.704758 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm"] Sep 11 19:39:49.706550 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:39:49.706529 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd2d22f_71ef_48d0_83ef_dd6ca3a2d3e4.slice/crio-d9042e71bd405e0ecd243eb64f59bc63709c6dc9ca67a0e964404a8821d34294 WatchSource:0}: Error finding container d9042e71bd405e0ecd243eb64f59bc63709c6dc9ca67a0e964404a8821d34294: Status 404 returned error can't find the container with id d9042e71bd405e0ecd243eb64f59bc63709c6dc9ca67a0e964404a8821d34294 Sep 11 19:39:49.708350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:49.708331 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:39:49.772790 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:49.772751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" event={"ID":"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4","Type":"ContainerStarted","Data":"d9042e71bd405e0ecd243eb64f59bc63709c6dc9ca67a0e964404a8821d34294"} Sep 11 19:39:50.788007 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:50.787947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:50.882522 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:50.882475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:50.982975 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:50.982926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:39:52.791254 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:52.791220 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" event={"ID":"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4","Type":"ContainerStarted","Data":"3d7093c276298c9ed45db95a8be3cfa621ed8746d82e35f341fad5bd3493c3bc"} Sep 11 19:39:53.797048 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:53.797018 2579 generic.go:356] "Generic (PLEG): container finished" podID="3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4" containerID="3d7093c276298c9ed45db95a8be3cfa621ed8746d82e35f341fad5bd3493c3bc" exitCode=0 Sep 11 19:39:53.797452 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:53.797080 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" event={"ID":"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4","Type":"ContainerDied","Data":"3d7093c276298c9ed45db95a8be3cfa621ed8746d82e35f341fad5bd3493c3bc"} Sep 11 19:39:54.803406 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:54.803370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" event={"ID":"3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4","Type":"ContainerStarted","Data":"06d5680c19d8c47e0392fbfc53923058d056abac164660887ea2e3e120fd3c37"} Sep 11 19:39:54.803805 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:54.803458 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:39:54.828380 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:39:54.828326 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" podStartSLOduration=31.401531113 podStartE2EDuration="35.828309323s" totalImagesPullingTime="3.421443309s" totalInitContainerRuntime="1.005334901s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:39:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:39:54.825705503 +0000 UTC m=+627.959018082" watchObservedRunningTime="2026-09-11 19:39:54.828309323 +0000 UTC m=+627.961621971" Sep 11 19:40:05.194065 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:05.194026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:05.257571 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:05.257525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:05.809885 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:05.809852 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-755nm" Sep 11 19:40:09.122445 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:09.122405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:09.230617 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:09.230559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:09.529713 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:09.529631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:09.646383 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:09.646348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:10.012102 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:10.012068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:10.352352 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:10.352318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:11.256055 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:11.256005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:11.356520 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:11.356491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:11.747606 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:11.747578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:11.824578 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:11.824545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:12.767743 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:12.767708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:14.430617 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.430574 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh"] Sep 11 19:40:14.440540 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.440512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.451304 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.451273 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 19:40:14.451304 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.451291 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 19:40:14.451503 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.451306 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 19:40:14.452353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.452335 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 19:40:14.452473 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.452336 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4qj94\"" Sep 11 19:40:14.452473 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.452342 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 19:40:14.464710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.464677 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh"] Sep 11 19:40:14.520001 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.519960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d940027e-7cb3-4d02-9b6c-bf0a111557ac-metrics-cert\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.520001 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.519998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2rt\" (UniqueName: \"kubernetes.io/projected/d940027e-7cb3-4d02-9b6c-bf0a111557ac-kube-api-access-7g2rt\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.520225 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.520076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d940027e-7cb3-4d02-9b6c-bf0a111557ac-cert\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.520225 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.520149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d940027e-7cb3-4d02-9b6c-bf0a111557ac-manager-config\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.621306 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.621265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d940027e-7cb3-4d02-9b6c-bf0a111557ac-manager-config\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.621306 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.621309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d940027e-7cb3-4d02-9b6c-bf0a111557ac-metrics-cert\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.621515 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.621327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2rt\" (UniqueName: \"kubernetes.io/projected/d940027e-7cb3-4d02-9b6c-bf0a111557ac-kube-api-access-7g2rt\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.621515 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.621372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d940027e-7cb3-4d02-9b6c-bf0a111557ac-cert\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.622061 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.622030 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d940027e-7cb3-4d02-9b6c-bf0a111557ac-manager-config\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.624106 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.624085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d940027e-7cb3-4d02-9b6c-bf0a111557ac-metrics-cert\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.624224 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.624144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d940027e-7cb3-4d02-9b6c-bf0a111557ac-cert\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.631572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.631548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2rt\" (UniqueName: \"kubernetes.io/projected/d940027e-7cb3-4d02-9b6c-bf0a111557ac-kube-api-access-7g2rt\") pod \"lws-controller-manager-f6d84646c-xdcdh\" (UID: \"d940027e-7cb3-4d02-9b6c-bf0a111557ac\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:14.750434 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:14.750340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:15.021449 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:15.021421 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh"] Sep 11 19:40:15.024192 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:40:15.024164 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd940027e_7cb3_4d02_9b6c_bf0a111557ac.slice/crio-5d718aa5924f5c5e17c2c6132182f6c7d1ecbaf8675c90a56c0fdcc10090cd24 WatchSource:0}: Error finding container 5d718aa5924f5c5e17c2c6132182f6c7d1ecbaf8675c90a56c0fdcc10090cd24: Status 404 returned error can't find the container with id 5d718aa5924f5c5e17c2c6132182f6c7d1ecbaf8675c90a56c0fdcc10090cd24 Sep 11 19:40:15.900385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:15.900346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" event={"ID":"d940027e-7cb3-4d02-9b6c-bf0a111557ac","Type":"ContainerStarted","Data":"5d718aa5924f5c5e17c2c6132182f6c7d1ecbaf8675c90a56c0fdcc10090cd24"} Sep 11 19:40:17.914382 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:17.914288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" event={"ID":"d940027e-7cb3-4d02-9b6c-bf0a111557ac","Type":"ContainerStarted","Data":"598e1bd9bb92988924f33cef3892da48182683e89accf42bb61f3695ac973ad7"} Sep 11 19:40:17.914744 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:17.914436 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:17.932270 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:17.932218 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" podStartSLOduration=1.328118766 podStartE2EDuration="3.932200268s" totalImagesPullingTime="2.604081502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:17.931654509 +0000 UTC m=+651.064967099" watchObservedRunningTime="2026-09-11 19:40:17.932200268 +0000 UTC m=+651.065512857" Sep 11 19:40:20.671515 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.671476 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hqd7z"] Sep 11 19:40:20.673588 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.673571 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:20.676253 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.676229 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 19:40:20.676376 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.676235 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8zgqf\"" Sep 11 19:40:20.683802 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.683777 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hqd7z"] Sep 11 19:40:20.701614 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.701580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:20.784326 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.784289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7d6\" (UniqueName: \"kubernetes.io/projected/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-kube-api-access-vs7d6\") pod \"odh-model-controller-7998c67dc4-hqd7z\" (UID: \"c4d63cf1-3033-4801-a4ac-31b25b4b3eca\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:20.784491 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.784341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-cert\") pod \"odh-model-controller-7998c67dc4-hqd7z\" (UID: \"c4d63cf1-3033-4801-a4ac-31b25b4b3eca\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:20.798180 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.798148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:20.885100 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.885065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7d6\" (UniqueName: \"kubernetes.io/projected/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-kube-api-access-vs7d6\") pod \"odh-model-controller-7998c67dc4-hqd7z\" (UID: \"c4d63cf1-3033-4801-a4ac-31b25b4b3eca\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:20.885284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.885135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-cert\") pod \"odh-model-controller-7998c67dc4-hqd7z\" (UID: \"c4d63cf1-3033-4801-a4ac-31b25b4b3eca\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:20.885284 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:40:20.885277 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 19:40:20.885365 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:40:20.885348 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-cert podName:c4d63cf1-3033-4801-a4ac-31b25b4b3eca nodeName:}" failed. No retries permitted until 2026-09-11 19:40:21.385329209 +0000 UTC m=+654.518641777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-cert") pod "odh-model-controller-7998c67dc4-hqd7z" (UID: "c4d63cf1-3033-4801-a4ac-31b25b4b3eca") : secret "odh-model-controller-webhook-cert" not found Sep 11 19:40:20.895358 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.895327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7d6\" (UniqueName: \"kubernetes.io/projected/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-kube-api-access-vs7d6\") pod \"odh-model-controller-7998c67dc4-hqd7z\" (UID: \"c4d63cf1-3033-4801-a4ac-31b25b4b3eca\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:20.898283 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:20.898256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:21.070646 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.070618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:21.249807 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.249775 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sl7f4"] Sep 11 19:40:21.252196 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.252178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:21.255393 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.255359 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 19:40:21.266385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.266360 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sl7f4"] Sep 11 19:40:21.279270 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.279238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:21.373711 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.373632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:21.389311 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.389279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-cert\") pod \"odh-model-controller-7998c67dc4-hqd7z\" (UID: \"c4d63cf1-3033-4801-a4ac-31b25b4b3eca\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:21.389472 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.389314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdcz\" (UniqueName: \"kubernetes.io/projected/1b238d23-1b8b-4958-b502-9a6a91c8be9a-kube-api-access-qrdcz\") pod \"model-serving-api-54d89bd5fc-sl7f4\" (UID: \"1b238d23-1b8b-4958-b502-9a6a91c8be9a\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:21.389472 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.389372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b238d23-1b8b-4958-b502-9a6a91c8be9a-tls-certs\") pod \"model-serving-api-54d89bd5fc-sl7f4\" (UID: \"1b238d23-1b8b-4958-b502-9a6a91c8be9a\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:21.392162 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.392129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4d63cf1-3033-4801-a4ac-31b25b4b3eca-cert\") pod \"odh-model-controller-7998c67dc4-hqd7z\" (UID: \"c4d63cf1-3033-4801-a4ac-31b25b4b3eca\") " pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:21.490640 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.490598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b238d23-1b8b-4958-b502-9a6a91c8be9a-tls-certs\") pod \"model-serving-api-54d89bd5fc-sl7f4\" (UID: \"1b238d23-1b8b-4958-b502-9a6a91c8be9a\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:21.490877 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.490697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdcz\" (UniqueName: \"kubernetes.io/projected/1b238d23-1b8b-4958-b502-9a6a91c8be9a-kube-api-access-qrdcz\") pod \"model-serving-api-54d89bd5fc-sl7f4\" (UID: \"1b238d23-1b8b-4958-b502-9a6a91c8be9a\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:21.493365 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.493344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b238d23-1b8b-4958-b502-9a6a91c8be9a-tls-certs\") pod \"model-serving-api-54d89bd5fc-sl7f4\" (UID: \"1b238d23-1b8b-4958-b502-9a6a91c8be9a\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:21.500262 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.500238 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdcz\" (UniqueName: \"kubernetes.io/projected/1b238d23-1b8b-4958-b502-9a6a91c8be9a-kube-api-access-qrdcz\") pod \"model-serving-api-54d89bd5fc-sl7f4\" (UID: \"1b238d23-1b8b-4958-b502-9a6a91c8be9a\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:21.562590 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.562555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:21.585021 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.584994 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:21.839699 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.839672 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sl7f4"] Sep 11 19:40:21.842273 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:40:21.842242 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b238d23_1b8b_4958_b502_9a6a91c8be9a.slice/crio-88d92b90af2f228b0a52ce206c5b77c047c285fa5db66589ad861abc772c19fa WatchSource:0}: Error finding container 88d92b90af2f228b0a52ce206c5b77c047c285fa5db66589ad861abc772c19fa: Status 404 returned error can't find the container with id 88d92b90af2f228b0a52ce206c5b77c047c285fa5db66589ad861abc772c19fa Sep 11 19:40:21.867625 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.867594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hqd7z"] Sep 11 19:40:21.870566 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:40:21.870539 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d63cf1_3033_4801_a4ac_31b25b4b3eca.slice/crio-34d95df99968d891a424460b69387aa115813c4e4db21e792e2997911b70e36f WatchSource:0}: Error finding container 34d95df99968d891a424460b69387aa115813c4e4db21e792e2997911b70e36f: Status 404 returned error can't find the container with id 34d95df99968d891a424460b69387aa115813c4e4db21e792e2997911b70e36f Sep 11 19:40:21.932770 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.932670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" event={"ID":"c4d63cf1-3033-4801-a4ac-31b25b4b3eca","Type":"ContainerStarted","Data":"34d95df99968d891a424460b69387aa115813c4e4db21e792e2997911b70e36f"} Sep 11 19:40:21.933884 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:21.933862 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" event={"ID":"1b238d23-1b8b-4958-b502-9a6a91c8be9a","Type":"ContainerStarted","Data":"88d92b90af2f228b0a52ce206c5b77c047c285fa5db66589ad861abc772c19fa"} Sep 11 19:40:22.200537 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.200440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:22.308119 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.308083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:22.978844 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.978797 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2"] Sep 11 19:40:22.982895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.982869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:22.987329 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.987272 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 19:40:22.988190 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.987893 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 19:40:22.990612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.988464 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-w5ql9\"" Sep 11 19:40:22.990612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.988537 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 19:40:22.995078 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:22.995044 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2"] Sep 11 19:40:23.013527 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.013494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:23.107404 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.107582 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.107582 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.107582 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.107582 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.107817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.107817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.107817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.107817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.107714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w8hh\" (UniqueName: \"kubernetes.io/projected/6dc26d31-389a-4c3f-856d-38c53f4d42dc-kube-api-access-2w8hh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.115041 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.115011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:23.208364 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.208325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.208556 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.208380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.208556 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.208427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.208956 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.208935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.209420 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.209391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.209541 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.209451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.209541 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.209477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.209541 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.209514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.209711 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.209557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w8hh\" (UniqueName: \"kubernetes.io/projected/6dc26d31-389a-4c3f-856d-38c53f4d42dc-kube-api-access-2w8hh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.209711 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.209626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.210393 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.210353 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.210506 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.210429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.210571 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.210502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.211015 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.210968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.215008 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.214949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.215809 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.215783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.219918 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.219893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc26d31-389a-4c3f-856d-38c53f4d42dc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.223405 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.223378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w8hh\" (UniqueName: \"kubernetes.io/projected/6dc26d31-389a-4c3f-856d-38c53f4d42dc-kube-api-access-2w8hh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2\" (UID: \"6dc26d31-389a-4c3f-856d-38c53f4d42dc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.302479 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.302443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:23.364570 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.364528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:23.613982 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:23.613688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2"] Sep 11 19:40:24.519387 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:40:24.519347 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc26d31_389a_4c3f_856d_38c53f4d42dc.slice/crio-8227f3c16efef9b4a5b96fa86318e8150c2d845520a7f50984124743e63753a9 WatchSource:0}: Error finding container 8227f3c16efef9b4a5b96fa86318e8150c2d845520a7f50984124743e63753a9: Status 404 returned error can't find the container with id 8227f3c16efef9b4a5b96fa86318e8150c2d845520a7f50984124743e63753a9 Sep 11 19:40:24.953868 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:24.953807 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" event={"ID":"6dc26d31-389a-4c3f-856d-38c53f4d42dc","Type":"ContainerStarted","Data":"8227f3c16efef9b4a5b96fa86318e8150c2d845520a7f50984124743e63753a9"} Sep 11 19:40:25.962975 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:25.962936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" event={"ID":"c4d63cf1-3033-4801-a4ac-31b25b4b3eca","Type":"ContainerStarted","Data":"ac832c193bac5db4690e076ed9edb71d6a05a6717d9904c0ddb35347b86676b0"} Sep 11 19:40:25.964213 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:25.964191 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:25.967980 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:25.967949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" event={"ID":"1b238d23-1b8b-4958-b502-9a6a91c8be9a","Type":"ContainerStarted","Data":"296a427f2f7ea9026b789fac51362f0ab164d67730b54fac97561203a5468bed"} Sep 11 19:40:26.993763 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:26.993705 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" podStartSLOduration=1.973824155 podStartE2EDuration="5.993681984s" totalImagesPullingTime="4.019857829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:26.993021007 +0000 UTC m=+660.126333596" watchObservedRunningTime="2026-09-11 19:40:26.993681984 +0000 UTC m=+660.126994586" Sep 11 19:40:26.995490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:26.995443 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" podStartSLOduration=2.990661759 podStartE2EDuration="6.995424622s" totalImagesPullingTime="4.004762863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:25.98249643 +0000 UTC m=+659.115809035" watchObservedRunningTime="2026-09-11 19:40:26.995424622 +0000 UTC m=+660.128737213" Sep 11 19:40:27.380041 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:27.380003 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:40:27.380177 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:27.380095 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:40:27.380177 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:27.380144 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:40:27.980441 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:27.980401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" event={"ID":"6dc26d31-389a-4c3f-856d-38c53f4d42dc","Type":"ContainerStarted","Data":"7cbb87c10175f7332d1c1c9be5bd76c477cf10154e516a85f3d6a89258c3d8ed"} Sep 11 19:40:27.980718 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:27.980702 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:28.008142 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.008078 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" podStartSLOduration=3.151352485 podStartE2EDuration="6.008056254s" totalImagesPullingTime="2.856703769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:28.005970247 +0000 UTC m=+661.139282828" watchObservedRunningTime="2026-09-11 19:40:28.008056254 +0000 UTC m=+661.141368843" Sep 11 19:40:28.242194 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.242109 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k"] Sep 11 19:40:28.245030 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.245003 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.251083 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.251056 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 19:40:28.251279 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.251260 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 19:40:28.251545 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.251529 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-p95qw\"" Sep 11 19:40:28.253001 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.252966 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 19:40:28.263706 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.263675 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k"] Sep 11 19:40:28.303588 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.303555 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:28.309103 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.309075 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:28.357385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.357341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.357545 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.357399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpmjx\" (UniqueName: \"kubernetes.io/projected/c8adb287-02d5-487c-8a6b-7e23bc2af729-kube-api-access-xpmjx\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.357545 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.357468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.357545 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.357515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.357702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.357553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8adb287-02d5-487c-8a6b-7e23bc2af729-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.357702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.357594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.357702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.357627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.458855 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.458786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.459052 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.458892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.459052 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.458923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpmjx\" (UniqueName: \"kubernetes.io/projected/c8adb287-02d5-487c-8a6b-7e23bc2af729-kube-api-access-xpmjx\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.459052 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.458942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.459052 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.458966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.459342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.459103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8adb287-02d5-487c-8a6b-7e23bc2af729-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.459342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.459172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.459652 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.459627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.462137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.462107 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8adb287-02d5-487c-8a6b-7e23bc2af729-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.462367 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.462350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.462417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.462358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.462417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.462394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.467892 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.467859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8adb287-02d5-487c-8a6b-7e23bc2af729-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.468094 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.468079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpmjx\" (UniqueName: \"kubernetes.io/projected/c8adb287-02d5-487c-8a6b-7e23bc2af729-kube-api-access-xpmjx\") pod \"istiod-openshift-gateway-5677ccb49b-lxt9k\" (UID: \"c8adb287-02d5-487c-8a6b-7e23bc2af729\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.557155 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.557118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:28.842687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.842576 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k"] Sep 11 19:40:28.845148 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:40:28.845109 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8adb287_02d5_487c_8a6b_7e23bc2af729.slice/crio-4a7ea2d90b55c70dba84972c12f534e5ba61f55883203a500016c9ff0ea083cd WatchSource:0}: Error finding container 4a7ea2d90b55c70dba84972c12f534e5ba61f55883203a500016c9ff0ea083cd: Status 404 returned error can't find the container with id 4a7ea2d90b55c70dba84972c12f534e5ba61f55883203a500016c9ff0ea083cd Sep 11 19:40:28.921137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.921103 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-xdcdh" Sep 11 19:40:28.998546 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.996349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" event={"ID":"c8adb287-02d5-487c-8a6b-7e23bc2af729","Type":"ContainerStarted","Data":"4a7ea2d90b55c70dba84972c12f534e5ba61f55883203a500016c9ff0ea083cd"} Sep 11 19:40:28.998546 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:28.997282 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:29.002847 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:29.000131 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2" Sep 11 19:40:30.477750 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:30.477676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:30.613461 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:30.613427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:30.682275 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:30.682243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:31.419670 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:31.419635 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:40:31.419772 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:31.419699 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:40:31.632917 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:31.632884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:31.956714 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:31.956678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:32.015657 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:32.015613 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" event={"ID":"c8adb287-02d5-487c-8a6b-7e23bc2af729","Type":"ContainerStarted","Data":"2df6a0b9052d1658a3638a78596407f6c8c95f51753b58981b560e1183a2977a"} Sep 11 19:40:32.015959 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:32.015937 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:32.017560 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:32.017536 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-lxt9k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 19:40:32.017670 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:32.017583 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" podUID="c8adb287-02d5-487c-8a6b-7e23bc2af729" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 19:40:32.077638 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:32.077108 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" podStartSLOduration=1.5056900880000001 podStartE2EDuration="4.077083749s" totalImagesPullingTime="2.571393661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:32.073857435 +0000 UTC m=+665.207170021" watchObservedRunningTime="2026-09-11 19:40:32.077083749 +0000 UTC m=+665.210396339" Sep 11 19:40:32.085251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:32.085203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:33.021858 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:33.021799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lxt9k" Sep 11 19:40:35.492217 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:35.492179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:35.589768 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:35.589738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:40:37.983006 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:37.982972 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hqd7z" Sep 11 19:40:39.000409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:39.000379 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-sl7f4" Sep 11 19:40:48.801172 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:40:48.801131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:21.561309 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:21.561238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:21.655638 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:21.655605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:35.437993 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:35.437960 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:36.230121 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:36.230091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:38.947604 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:38.947573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:39.112229 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:39.112174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:39.645679 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:39.645647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:40.037560 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:40.037530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:40.540064 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:40.540031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:41.011634 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:41.011539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:41.172470 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:41.172440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:43.870745 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:43.870711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:43.964680 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:43.964649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:46.049115 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:46.049078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:46.295924 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:46.295881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:49.116847 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:49.116789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:49.216516 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:49.216486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:49.466885 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:49.466762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:49.562647 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:49.562611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:53.426187 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:53.426148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:53.741274 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:53.741173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:54.850916 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:54.850883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:54.916598 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:54.916562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:55.052579 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:55.052547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:55.123843 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:55.123740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:55.242237 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:55.242202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:41:55.349011 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:41:55.348976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:01.830359 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:01.830284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:01.926525 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:01.926490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:04.222571 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:04.222524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:04.299383 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:04.299349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:09.014192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:09.014150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:09.113777 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:09.113742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:09.956945 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:09.956905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:10.033447 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:10.033418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:14.764044 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:14.764011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:14.847553 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:14.847515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:15.160457 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:15.160426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:15.264129 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:15.264081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:15.572644 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:15.572612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:15.646228 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:15.646191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:21.454593 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:21.454558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:21.558051 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:21.553123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:23.933019 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:23.932980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:24.032285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:24.032252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:30.483735 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:30.483698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:30.584133 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:30.584091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:33.756204 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.756173 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z"] Sep 11 19:42:33.759555 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.759534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.777011 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.776981 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2h9m9\"" Sep 11 19:42:33.778753 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.778729 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z"] Sep 11 19:42:33.784812 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.784780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf996906-b8e6-4323-af98-f888584e2d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.784983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.784844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.784983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.784871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf996906-b8e6-4323-af98-f888584e2d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.784983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.784908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.784983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.784931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.785159 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.784986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.785159 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.785009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.785159 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.785105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbz7m\" (UniqueName: \"kubernetes.io/projected/bf996906-b8e6-4323-af98-f888584e2d1c-kube-api-access-rbz7m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.785159 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.785152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf996906-b8e6-4323-af98-f888584e2d1c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.797319 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.797271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:33.885708 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.885676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.885708 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.885711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf996906-b8e6-4323-af98-f888584e2d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.885952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.885737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.885952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.885866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.885952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.885915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886118 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886165 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbz7m\" (UniqueName: \"kubernetes.io/projected/bf996906-b8e6-4323-af98-f888584e2d1c-kube-api-access-rbz7m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886165 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf996906-b8e6-4323-af98-f888584e2d1c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886259 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886259 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886360 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf996906-b8e6-4323-af98-f888584e2d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886557 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf996906-b8e6-4323-af98-f888584e2d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886681 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.886735 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.886680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.888716 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.888690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf996906-b8e6-4323-af98-f888584e2d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.888813 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.888797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf996906-b8e6-4323-af98-f888584e2d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.894859 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.894813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbz7m\" (UniqueName: \"kubernetes.io/projected/bf996906-b8e6-4323-af98-f888584e2d1c-kube-api-access-rbz7m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:33.895103 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:33.895085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf996906-b8e6-4323-af98-f888584e2d1c-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fvg7z\" (UID: \"bf996906-b8e6-4323-af98-f888584e2d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:34.017374 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.017271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:34.070343 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.070314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:34.355136 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.355094 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z"] Sep 11 19:42:34.359802 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:42:34.359770 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf996906_b8e6_4323_af98_f888584e2d1c.slice/crio-7371b6126298ee83b3be21758198ed8df2723ac4b127ea7aaf697654d742792b WatchSource:0}: Error finding container 7371b6126298ee83b3be21758198ed8df2723ac4b127ea7aaf697654d742792b: Status 404 returned error can't find the container with id 7371b6126298ee83b3be21758198ed8df2723ac4b127ea7aaf697654d742792b Sep 11 19:42:34.362141 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.362107 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:42:34.362220 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.362176 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:42:34.362220 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.362205 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:42:34.588180 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.588145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" event={"ID":"bf996906-b8e6-4323-af98-f888584e2d1c","Type":"ContainerStarted","Data":"1cd4d5dd23fe55f65ce096f84c7ee3397534726487c8fd481aa77cfba229337a"} Sep 11 19:42:34.588356 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.588202 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" event={"ID":"bf996906-b8e6-4323-af98-f888584e2d1c","Type":"ContainerStarted","Data":"7371b6126298ee83b3be21758198ed8df2723ac4b127ea7aaf697654d742792b"} Sep 11 19:42:34.611335 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:34.611222 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" podStartSLOduration=1.611203208 podStartE2EDuration="1.611203208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:42:34.609135858 +0000 UTC m=+787.742448447" watchObservedRunningTime="2026-09-11 19:42:34.611203208 +0000 UTC m=+787.744515798" Sep 11 19:42:35.070781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:35.070746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:35.075764 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:35.075739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:35.564334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:35.564286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:35.593818 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:35.593784 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:35.594948 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:35.594928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:42:35.665088 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:35.665045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:35.771028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:35.771001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:36.413637 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:36.413605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:36.536014 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:36.535981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:51.651667 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:51.651543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:51.740273 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:51.740215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:42:52.081926 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:42:52.081896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:01.970710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:01.970675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:02.192325 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:02.192281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:08.452928 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:08.452891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:08.550560 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:08.550523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:08.624749 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:08.624714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:08.703597 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:08.703516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:08.795387 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:08.795349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:12.079549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:12.079495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:12.177604 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:12.177549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:26.200053 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:26.200006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:26.300090 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:26.300054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:28.923653 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:28.923605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:29.024353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:29.024323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:41.311908 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:41.311866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:41.410021 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:41.409987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:42.749102 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:42.749070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:42.807774 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:42.807738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:42.928298 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:42.928254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:43.208783 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.208742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:43.420578 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.420529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:43.514110 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.513991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:43.673955 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.673918 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-z2ht6"] Sep 11 19:43:43.677559 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.677537 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:43.680498 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.680469 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-68flw\"" Sep 11 19:43:43.680649 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.680480 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 19:43:43.689183 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.689158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-z2ht6"] Sep 11 19:43:43.735509 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.735466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p9xs\" (UniqueName: \"kubernetes.io/projected/9d9a0f88-e534-415a-bfea-a3e605edc458-kube-api-access-9p9xs\") pod \"payload-processing-5b54b7fbb7-z2ht6\" (UID: \"9d9a0f88-e534-415a-bfea-a3e605edc458\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:43.735705 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.735519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9a0f88-e534-415a-bfea-a3e605edc458-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-z2ht6\" (UID: \"9d9a0f88-e534-415a-bfea-a3e605edc458\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:43.836365 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.836315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p9xs\" (UniqueName: \"kubernetes.io/projected/9d9a0f88-e534-415a-bfea-a3e605edc458-kube-api-access-9p9xs\") pod \"payload-processing-5b54b7fbb7-z2ht6\" (UID: \"9d9a0f88-e534-415a-bfea-a3e605edc458\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:43.836365 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.836374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9a0f88-e534-415a-bfea-a3e605edc458-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-z2ht6\" (UID: \"9d9a0f88-e534-415a-bfea-a3e605edc458\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:43.837096 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.837077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9a0f88-e534-415a-bfea-a3e605edc458-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-z2ht6\" (UID: \"9d9a0f88-e534-415a-bfea-a3e605edc458\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:43.845010 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.844982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p9xs\" (UniqueName: \"kubernetes.io/projected/9d9a0f88-e534-415a-bfea-a3e605edc458-kube-api-access-9p9xs\") pod \"payload-processing-5b54b7fbb7-z2ht6\" (UID: \"9d9a0f88-e534-415a-bfea-a3e605edc458\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:43.990616 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:43.990581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:44.266453 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:44.266354 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-z2ht6"] Sep 11 19:43:44.294880 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:43:44.294810 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9a0f88_e534_415a_bfea_a3e605edc458.slice/crio-a2bee480f6dedafa53fe518660b6a7ab529b8091672c791ec45bd32bbb026435 WatchSource:0}: Error finding container a2bee480f6dedafa53fe518660b6a7ab529b8091672c791ec45bd32bbb026435: Status 404 returned error can't find the container with id a2bee480f6dedafa53fe518660b6a7ab529b8091672c791ec45bd32bbb026435 Sep 11 19:43:44.927067 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:44.927028 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" event={"ID":"9d9a0f88-e534-415a-bfea-a3e605edc458","Type":"ContainerStarted","Data":"a2bee480f6dedafa53fe518660b6a7ab529b8091672c791ec45bd32bbb026435"} Sep 11 19:43:45.935035 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:45.934995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" event={"ID":"9d9a0f88-e534-415a-bfea-a3e605edc458","Type":"ContainerStarted","Data":"eb06b2ec6a190a1c6ba5bcf09cd1dfb30471e35bea87a0137c64609579415331"} Sep 11 19:43:45.935420 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:45.935097 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:45.955787 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:45.954439 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" podStartSLOduration=1.5268441780000002 podStartE2EDuration="2.954417259s" totalImagesPullingTime="1.427573081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:43:45.953640952 +0000 UTC m=+859.086953563" watchObservedRunningTime="2026-09-11 19:43:45.954417259 +0000 UTC m=+859.087729849" Sep 11 19:43:51.635048 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:51.634988 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fvg7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 19:43:51.635557 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:51.635062 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" podUID="bf996906-b8e6-4323-af98-f888584e2d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 19:43:52.968368 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:52.968288 2579 generic.go:356] "Generic (PLEG): container finished" podID="bf996906-b8e6-4323-af98-f888584e2d1c" containerID="1cd4d5dd23fe55f65ce096f84c7ee3397534726487c8fd481aa77cfba229337a" exitCode=0 Sep 11 19:43:52.968923 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:52.968362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" event={"ID":"bf996906-b8e6-4323-af98-f888584e2d1c","Type":"ContainerDied","Data":"1cd4d5dd23fe55f65ce096f84c7ee3397534726487c8fd481aa77cfba229337a"} Sep 11 19:43:52.968923 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:52.968909 2579 scope.go:117] "RemoveContainer" containerID="1cd4d5dd23fe55f65ce096f84c7ee3397534726487c8fd481aa77cfba229337a" Sep 11 19:43:52.970131 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:52.970093 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:43:52.970237 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:52.970184 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:43:52.970282 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:52.970233 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:43:53.902767 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:53.902731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:53.979916 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:53.978918 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" event={"ID":"bf996906-b8e6-4323-af98-f888584e2d1c","Type":"ContainerStarted","Data":"175aaca3107937cb7069646e369f5f05810c19b00eefc287c1277d82da8cb6b2"} Sep 11 19:43:54.070482 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:54.070437 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:43:54.070895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:54.070868 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fvg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 11 19:43:54.071124 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:54.071097 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" podUID="bf996906-b8e6-4323-af98-f888584e2d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 11 19:43:55.071394 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:55.070922 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fvg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 11 19:43:55.071394 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:55.070993 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" podUID="bf996906-b8e6-4323-af98-f888584e2d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 11 19:43:55.287314 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:55.287285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:43:56.070875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:56.070837 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fvg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 11 19:43:56.071176 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:56.071145 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" podUID="bf996906-b8e6-4323-af98-f888584e2d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 11 19:43:56.942503 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:56.942475 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2ht6" Sep 11 19:43:57.074345 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:57.073841 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fvg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 11 19:43:57.074345 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:57.073918 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" podUID="bf996906-b8e6-4323-af98-f888584e2d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 11 19:43:58.079965 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:58.079927 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fvg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 11 19:43:58.080564 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:58.080531 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" podUID="bf996906-b8e6-4323-af98-f888584e2d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 11 19:43:59.071637 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:59.071595 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fvg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 11 19:43:59.071972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:59.071936 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" podUID="bf996906-b8e6-4323-af98-f888584e2d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 11 19:43:59.293926 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:43:59.293890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:00.182479 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:00.182451 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:44:00.182880 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:00.182860 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:44:00.183565 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:00.183548 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fvg7z" Sep 11 19:44:05.496309 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:05.496276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:05.596975 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:05.596870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:13.990656 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:13.990613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:21.987182 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:21.987094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:22.492982 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.492947 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs"] Sep 11 19:44:22.496247 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.496227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.499327 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.499299 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 19:44:22.500385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.500366 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 19:44:22.500385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.500375 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vwxjz\"" Sep 11 19:44:22.500543 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.500450 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 19:44:22.510683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.510660 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs"] Sep 11 19:44:22.549585 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.549556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:22.604200 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.604167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.604342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.604227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.604342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.604286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.604342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.604322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.604461 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.604342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvcwh\" (UniqueName: \"kubernetes.io/projected/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-kube-api-access-dvcwh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.604461 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.604366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705456 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705456 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705456 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvcwh\" (UniqueName: \"kubernetes.io/projected/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-kube-api-access-dvcwh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705456 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.705818 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.705795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.707722 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.707704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.708113 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.708093 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.714380 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.714349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvcwh\" (UniqueName: \"kubernetes.io/projected/e00b7b77-1dc2-4149-b9da-70d9e93b9cd0-kube-api-access-dvcwh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs\" (UID: \"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:22.806927 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:22.806891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:23.082589 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:23.082566 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs"] Sep 11 19:44:23.084769 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:44:23.084728 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00b7b77_1dc2_4149_b9da_70d9e93b9cd0.slice/crio-6b9c13d3ec48fc8e65d553211a8fcc73e6e691c33c653e6bbcf979f50cf20c68 WatchSource:0}: Error finding container 6b9c13d3ec48fc8e65d553211a8fcc73e6e691c33c653e6bbcf979f50cf20c68: Status 404 returned error can't find the container with id 6b9c13d3ec48fc8e65d553211a8fcc73e6e691c33c653e6bbcf979f50cf20c68 Sep 11 19:44:23.126888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:23.126856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" event={"ID":"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0","Type":"ContainerStarted","Data":"6b9c13d3ec48fc8e65d553211a8fcc73e6e691c33c653e6bbcf979f50cf20c68"} Sep 11 19:44:25.138943 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:25.138907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" event={"ID":"e00b7b77-1dc2-4149-b9da-70d9e93b9cd0","Type":"ContainerStarted","Data":"23ef83c86e85736709f32340d7b0f44c477c6dcbcd782383e8e44b57cc6840dd"} Sep 11 19:44:25.162754 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:25.162699 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" podStartSLOduration=2.172132549 podStartE2EDuration="3.162682025s" totalImagesPullingTime="990.549476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:25.160616076 +0000 UTC m=+898.293928664" watchObservedRunningTime="2026-09-11 19:44:25.162682025 +0000 UTC m=+898.295994613" Sep 11 19:44:27.362901 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:27.362865 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:44:27.364969 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:27.364946 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:44:27.760720 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:27.760606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:32.807277 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:32.807236 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:32.807803 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:32.807432 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:32.820420 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:32.820395 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:33.188586 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:33.188486 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs" Sep 11 19:44:33.867519 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:33.867484 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh"] Sep 11 19:44:33.872113 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:33.872088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:33.875276 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:33.875259 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 19:44:33.910298 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:33.910251 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh"] Sep 11 19:44:33.967117 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:33.967085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:34.014889 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.014846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.015074 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.014923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8b552c-b7ab-4bef-873c-418aaae81e3b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.015074 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.015038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.015188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.015074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.015188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.015129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.015274 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.015196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmtv\" (UniqueName: \"kubernetes.io/projected/fc8b552c-b7ab-4bef-873c-418aaae81e3b-kube-api-access-zlmtv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116056 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8b552c-b7ab-4bef-873c-418aaae81e3b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmtv\" (UniqueName: \"kubernetes.io/projected/fc8b552c-b7ab-4bef-873c-418aaae81e3b-kube-api-access-zlmtv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116521 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116521 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.116623 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.116581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.118625 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.118574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8b552c-b7ab-4bef-873c-418aaae81e3b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.118834 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.118807 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8b552c-b7ab-4bef-873c-418aaae81e3b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.127445 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.127421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmtv\" (UniqueName: \"kubernetes.io/projected/fc8b552c-b7ab-4bef-873c-418aaae81e3b-kube-api-access-zlmtv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh\" (UID: \"fc8b552c-b7ab-4bef-873c-418aaae81e3b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.182337 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.182304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:34.459154 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:34.459125 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh"] Sep 11 19:44:34.461917 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:44:34.461885 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8b552c_b7ab_4bef_873c_418aaae81e3b.slice/crio-cf08f531b2c81325e30398713a659cd8ae77777cba5c2cfdaf764ecd0ced2efd WatchSource:0}: Error finding container cf08f531b2c81325e30398713a659cd8ae77777cba5c2cfdaf764ecd0ced2efd: Status 404 returned error can't find the container with id cf08f531b2c81325e30398713a659cd8ae77777cba5c2cfdaf764ecd0ced2efd Sep 11 19:44:35.186602 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:35.186572 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" event={"ID":"fc8b552c-b7ab-4bef-873c-418aaae81e3b","Type":"ContainerStarted","Data":"a5141ce19e7a663853338c6f4e48a096b8c984348a2cdbd394356395dddb009a"} Sep 11 19:44:35.187035 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:35.186621 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" event={"ID":"fc8b552c-b7ab-4bef-873c-418aaae81e3b","Type":"ContainerStarted","Data":"cf08f531b2c81325e30398713a659cd8ae77777cba5c2cfdaf764ecd0ced2efd"} Sep 11 19:44:35.208302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:35.208246 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" podStartSLOduration=2.028166473 podStartE2EDuration="2.208228671s" totalImagesPullingTime="180.062198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:35.206752266 +0000 UTC m=+908.340064871" watchObservedRunningTime="2026-09-11 19:44:35.208228671 +0000 UTC m=+908.341541260" Sep 11 19:44:38.745801 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:38.745763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:38.812228 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:38.812189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:44.183231 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:44.183181 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:44.183659 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:44.183259 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:44.195787 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:44.195755 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:44.247185 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:44.247151 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh" Sep 11 19:44:52.019425 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:52.019392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:44:58.597549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:44:58.597516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:29.468475 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:29.468443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:29.579140 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:29.579111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:30.591711 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:30.591682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:30.686015 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:30.685986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:45.735132 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:45.735099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:45.818093 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:45.818055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:45.900442 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:45.900408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:46.097507 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.097465 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4"] Sep 11 19:46:46.101880 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.101854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.105112 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.105087 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a8ee95-openshift-default-dockercfg-2fdgv\"" Sep 11 19:46:46.137345 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.137307 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4"] Sep 11 19:46:46.154939 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.154893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-certs\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.155137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.154973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686e089b-fcff-45f8-8481-67d970a09e8e-istiod-ca-cert\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.155137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.155018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-istio-token\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.155137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.155054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-data\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.155137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.155093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686e089b-fcff-45f8-8481-67d970a09e8e-istio-podinfo\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.155310 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.155143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-credential-socket\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.155310 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.155170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjp5\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-kube-api-access-jjjp5\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.155310 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.155277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-envoy\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.155420 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.155327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-socket\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.216364 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.216325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:46.256172 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-socket\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.256379 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-certs\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.256379 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686e089b-fcff-45f8-8481-67d970a09e8e-istiod-ca-cert\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.256615 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-istio-token\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.256756 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-data\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.256756 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686e089b-fcff-45f8-8481-67d970a09e8e-istio-podinfo\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.256954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-credential-socket\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.256954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-socket\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.256954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjp5\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-kube-api-access-jjjp5\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.257131 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.256955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-envoy\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.257453 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.257182 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-credential-socket\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.257453 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.257387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686e089b-fcff-45f8-8481-67d970a09e8e-istiod-ca-cert\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.257777 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.257638 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-data\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.257956 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.257935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-certs\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.259451 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.259425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-envoy\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.259635 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.259614 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686e089b-fcff-45f8-8481-67d970a09e8e-istio-podinfo\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.277191 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.277160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-istio-token\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.277364 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.277209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjp5\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-kube-api-access-jjjp5\") pod \"e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.362218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.362108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:46.379116 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.379073 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t"] Sep 11 19:46:46.384506 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.384479 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.387421 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.387397 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a58a49-openshift-default-dockercfg-vtnd7\"" Sep 11 19:46:46.399142 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.399115 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t"] Sep 11 19:46:46.413033 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.412999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:46.458230 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:46.458385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29576270-8f07-4992-b746-550d5d05b2f5-istio-podinfo\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.458385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-envoy\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.458478 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-istio-token\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.458478 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmgw\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-kube-api-access-tpmgw\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.458547 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-certs\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.458616 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29576270-8f07-4992-b746-550d5d05b2f5-istiod-ca-cert\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.458616 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-socket\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.458616 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-credential-socket\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.458776 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.458648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-data\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.559642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-envoy\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.559724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-istio-token\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.559759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmgw\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-kube-api-access-tpmgw\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.559787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-certs\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.559897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29576270-8f07-4992-b746-550d5d05b2f5-istiod-ca-cert\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.559925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-socket\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.559947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-credential-socket\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.560003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-data\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.560218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.560053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29576270-8f07-4992-b746-550d5d05b2f5-istio-podinfo\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.562356 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.562314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-credential-socket\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.562520 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.562398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:46.562797 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.562693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-socket\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.562797 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.562708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29576270-8f07-4992-b746-550d5d05b2f5-istiod-ca-cert\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.563023 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.562995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-data\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.563087 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.563035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-certs\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.563378 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.563307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-envoy\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.565499 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.565469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29576270-8f07-4992-b746-550d5d05b2f5-istio-podinfo\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.572514 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.572460 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-istio-token\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.573426 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.573371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmgw\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-kube-api-access-tpmgw\") pod \"e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.697960 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.697872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:46.702915 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.702887 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4"] Sep 11 19:46:46.704964 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:46:46.704933 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686e089b_fcff_45f8_8481_67d970a09e8e.slice/crio-f10e5c972883ee142b281352972cd628abd16c46450299a66849dbe6ff703164 WatchSource:0}: Error finding container f10e5c972883ee142b281352972cd628abd16c46450299a66849dbe6ff703164: Status 404 returned error can't find the container with id f10e5c972883ee142b281352972cd628abd16c46450299a66849dbe6ff703164 Sep 11 19:46:46.708490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.708460 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:46:46.708577 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.708534 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:46:46.708577 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.708562 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:46:46.817618 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.817564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:46.842880 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.842802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" event={"ID":"686e089b-fcff-45f8-8481-67d970a09e8e","Type":"ContainerStarted","Data":"462851e3465c164a16247a7e9611da5e8a944eeeb4cc10406b481c2cf63c2385"} Sep 11 19:46:46.843085 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.842950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" event={"ID":"686e089b-fcff-45f8-8481-67d970a09e8e","Type":"ContainerStarted","Data":"f10e5c972883ee142b281352972cd628abd16c46450299a66849dbe6ff703164"} Sep 11 19:46:46.864057 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:46.863997 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" podStartSLOduration=0.863961322 podStartE2EDuration="863.961322ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:46.863077437 +0000 UTC m=+1039.996390062" watchObservedRunningTime="2026-09-11 19:46:46.863961322 +0000 UTC m=+1039.997273911" Sep 11 19:46:47.004799 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.004710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t"] Sep 11 19:46:47.009793 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:46:47.009760 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29576270_8f07_4992_b746_550d5d05b2f5.slice/crio-f740739962192a831569d18ba729968db36b3356b0806e74dc26817311e88c31 WatchSource:0}: Error finding container f740739962192a831569d18ba729968db36b3356b0806e74dc26817311e88c31: Status 404 returned error can't find the container with id f740739962192a831569d18ba729968db36b3356b0806e74dc26817311e88c31 Sep 11 19:46:47.012458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.012413 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:46:47.012552 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.012513 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:46:47.012592 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.012553 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:46:47.226759 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.226678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:47.377002 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.376966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:47.413893 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.413866 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:47.419630 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.419606 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:47.671019 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.670917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:47.849498 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.849467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" event={"ID":"29576270-8f07-4992-b746-550d5d05b2f5","Type":"ContainerStarted","Data":"8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4"} Sep 11 19:46:47.849949 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.849518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" event={"ID":"29576270-8f07-4992-b746-550d5d05b2f5","Type":"ContainerStarted","Data":"f740739962192a831569d18ba729968db36b3356b0806e74dc26817311e88c31"} Sep 11 19:46:47.849949 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.849723 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:47.850728 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.850711 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:46:47.878253 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:47.878192 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" podStartSLOduration=1.878170158 podStartE2EDuration="1.878170158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:47.877660578 +0000 UTC m=+1041.010973171" watchObservedRunningTime="2026-09-11 19:46:47.878170158 +0000 UTC m=+1041.011482747" Sep 11 19:46:48.698541 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:48.698496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:48.703972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:48.703946 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:48.855175 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:48.855137 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:48.855949 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:48.855928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:46:48.896155 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:48.896103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:49.020239 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:49.020115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:49.841135 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:49.841087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:49.928126 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:49.928097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:50.439098 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:50.439070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:50.571899 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:50.571862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:50.805059 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:50.805028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:51.441136 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.441096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:51.541996 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.541945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:51.563303 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.563261 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk"] Sep 11 19:46:51.567145 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.567124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:51.570016 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.569994 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-a8ee95\"" Sep 11 19:46:51.582439 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.582388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk"] Sep 11 19:46:51.635553 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.635518 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s"] Sep 11 19:46:51.639439 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.639417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:51.643028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.642992 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-a8ee95-dockercfg-w4r2z\"" Sep 11 19:46:51.653783 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.653754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s"] Sep 11 19:46:51.716938 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.715464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146408ba-5617-47ee-adba-3baf420d38d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s\" (UID: \"146408ba-5617-47ee-adba-3baf420d38d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:51.716938 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.715514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ac68d-4503-4374-b598-033d7687e0d1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk\" (UID: \"371ac68d-4503-4374-b598-033d7687e0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:51.716938 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.715551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmxf\" (UniqueName: \"kubernetes.io/projected/371ac68d-4503-4374-b598-033d7687e0d1-kube-api-access-mxmxf\") pod \"payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk\" (UID: \"371ac68d-4503-4374-b598-033d7687e0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:51.716938 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.715569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27594\" (UniqueName: \"kubernetes.io/projected/146408ba-5617-47ee-adba-3baf420d38d1-kube-api-access-27594\") pod \"payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s\" (UID: \"146408ba-5617-47ee-adba-3baf420d38d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:51.816761 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.816718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmxf\" (UniqueName: \"kubernetes.io/projected/371ac68d-4503-4374-b598-033d7687e0d1-kube-api-access-mxmxf\") pod \"payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk\" (UID: \"371ac68d-4503-4374-b598-033d7687e0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:51.816761 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.816776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27594\" (UniqueName: \"kubernetes.io/projected/146408ba-5617-47ee-adba-3baf420d38d1-kube-api-access-27594\") pod \"payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s\" (UID: \"146408ba-5617-47ee-adba-3baf420d38d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:51.817066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.816926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146408ba-5617-47ee-adba-3baf420d38d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s\" (UID: \"146408ba-5617-47ee-adba-3baf420d38d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:51.817066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.816988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ac68d-4503-4374-b598-033d7687e0d1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk\" (UID: \"371ac68d-4503-4374-b598-033d7687e0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:51.817941 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.817917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146408ba-5617-47ee-adba-3baf420d38d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s\" (UID: \"146408ba-5617-47ee-adba-3baf420d38d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:51.818022 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.817921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ac68d-4503-4374-b598-033d7687e0d1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk\" (UID: \"371ac68d-4503-4374-b598-033d7687e0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:51.828492 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.828459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmxf\" (UniqueName: \"kubernetes.io/projected/371ac68d-4503-4374-b598-033d7687e0d1-kube-api-access-mxmxf\") pod \"payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk\" (UID: \"371ac68d-4503-4374-b598-033d7687e0d1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:51.828595 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.828541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27594\" (UniqueName: \"kubernetes.io/projected/146408ba-5617-47ee-adba-3baf420d38d1-kube-api-access-27594\") pod \"payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s\" (UID: \"146408ba-5617-47ee-adba-3baf420d38d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:51.878397 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.878364 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:51.954593 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:51.954559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:52.015056 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.014684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:52.120765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.120734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:52.174850 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.174736 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk"] Sep 11 19:46:52.175590 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:46:52.175562 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371ac68d_4503_4374_b598_033d7687e0d1.slice/crio-b39685f71e61ed99bfcc27368cbe0451292730f9d3caf1c4ae1c95cf8e2d8d38 WatchSource:0}: Error finding container b39685f71e61ed99bfcc27368cbe0451292730f9d3caf1c4ae1c95cf8e2d8d38: Status 404 returned error can't find the container with id b39685f71e61ed99bfcc27368cbe0451292730f9d3caf1c4ae1c95cf8e2d8d38 Sep 11 19:46:52.296705 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.296678 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s"] Sep 11 19:46:52.298967 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:46:52.298939 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146408ba_5617_47ee_adba_3baf420d38d1.slice/crio-f97283886265087f089753ec20f7816159eb2437ad585e7191dabc8300097c90 WatchSource:0}: Error finding container f97283886265087f089753ec20f7816159eb2437ad585e7191dabc8300097c90: Status 404 returned error can't find the container with id f97283886265087f089753ec20f7816159eb2437ad585e7191dabc8300097c90 Sep 11 19:46:52.328577 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.328549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:52.446584 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.446550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:52.544868 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.544809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:52.760977 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.760943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:52.879076 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.878966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" event={"ID":"371ac68d-4503-4374-b598-033d7687e0d1","Type":"ContainerStarted","Data":"3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e"} Sep 11 19:46:52.879231 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.879186 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:46:52.879290 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.879229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" event={"ID":"371ac68d-4503-4374-b598-033d7687e0d1","Type":"ContainerStarted","Data":"b39685f71e61ed99bfcc27368cbe0451292730f9d3caf1c4ae1c95cf8e2d8d38"} Sep 11 19:46:52.881018 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.880988 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" event={"ID":"146408ba-5617-47ee-adba-3baf420d38d1","Type":"ContainerStarted","Data":"20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d"} Sep 11 19:46:52.881164 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.881036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" event={"ID":"146408ba-5617-47ee-adba-3baf420d38d1","Type":"ContainerStarted","Data":"f97283886265087f089753ec20f7816159eb2437ad585e7191dabc8300097c90"} Sep 11 19:46:52.881164 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.881106 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:46:52.914132 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:52.914080 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" podStartSLOduration=1.914064426 podStartE2EDuration="1.914064426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:52.911776337 +0000 UTC m=+1046.045088938" watchObservedRunningTime="2026-09-11 19:46:52.914064426 +0000 UTC m=+1046.047377015" Sep 11 19:46:53.000919 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.000852 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" podStartSLOduration=2.000813905 podStartE2EDuration="2.000813905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:52.999207195 +0000 UTC m=+1046.132519814" watchObservedRunningTime="2026-09-11 19:46:53.000813905 +0000 UTC m=+1046.134126496" Sep 11 19:46:53.033096 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.033066 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx"] Sep 11 19:46:53.036361 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.036342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.039918 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.039895 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a58a49\"" Sep 11 19:46:53.040053 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.039924 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a58a49-dockercfg-555cb\"" Sep 11 19:46:53.051916 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.051888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:53.053292 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.053269 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx"] Sep 11 19:46:53.130383 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.130309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e9ad1-be41-4c23-92e0-adec195d8421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx\" (UID: \"8f8e9ad1-be41-4c23-92e0-adec195d8421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.130383 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.130348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swj9\" (UniqueName: \"kubernetes.io/projected/8f8e9ad1-be41-4c23-92e0-adec195d8421-kube-api-access-7swj9\") pod \"payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx\" (UID: \"8f8e9ad1-be41-4c23-92e0-adec195d8421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.231682 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.231643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swj9\" (UniqueName: \"kubernetes.io/projected/8f8e9ad1-be41-4c23-92e0-adec195d8421-kube-api-access-7swj9\") pod \"payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx\" (UID: \"8f8e9ad1-be41-4c23-92e0-adec195d8421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.231875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.231790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e9ad1-be41-4c23-92e0-adec195d8421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx\" (UID: \"8f8e9ad1-be41-4c23-92e0-adec195d8421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.232538 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.232515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e9ad1-be41-4c23-92e0-adec195d8421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx\" (UID: \"8f8e9ad1-be41-4c23-92e0-adec195d8421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.241817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.241790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swj9\" (UniqueName: \"kubernetes.io/projected/8f8e9ad1-be41-4c23-92e0-adec195d8421-kube-api-access-7swj9\") pod \"payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx\" (UID: \"8f8e9ad1-be41-4c23-92e0-adec195d8421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.347496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.347460 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.463061 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.463030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:53.632942 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.632908 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx"] Sep 11 19:46:53.634726 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:46:53.634698 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8e9ad1_be41_4c23_92e0_adec195d8421.slice/crio-ae87bdc92a6622afb92fa1b834921a7e4038568d8435374e61e649d970395c1d WatchSource:0}: Error finding container ae87bdc92a6622afb92fa1b834921a7e4038568d8435374e61e649d970395c1d: Status 404 returned error can't find the container with id ae87bdc92a6622afb92fa1b834921a7e4038568d8435374e61e649d970395c1d Sep 11 19:46:53.890257 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.890217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" event={"ID":"8f8e9ad1-be41-4c23-92e0-adec195d8421","Type":"ContainerStarted","Data":"1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98"} Sep 11 19:46:53.890465 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.890283 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" event={"ID":"8f8e9ad1-be41-4c23-92e0-adec195d8421","Type":"ContainerStarted","Data":"ae87bdc92a6622afb92fa1b834921a7e4038568d8435374e61e649d970395c1d"} Sep 11 19:46:53.890543 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.890523 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:46:53.914318 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:53.914217 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" podStartSLOduration=1.9142012259999999 podStartE2EDuration="1.914201226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:53.911118744 +0000 UTC m=+1047.044431335" watchObservedRunningTime="2026-09-11 19:46:53.914201226 +0000 UTC m=+1047.047513817" Sep 11 19:46:54.249513 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:54.249405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:54.592987 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:54.592942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:54.743868 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:54.743807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:54.910894 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:54.910770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:55.726188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:55.726147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:55.923954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:55.923912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:56.147969 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:56.147931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:56.635204 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:56.635160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:56.730943 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:56.730912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:56.947955 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:56.947869 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r"] Sep 11 19:46:56.953006 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:56.952946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:56.957478 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:56.957447 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4b032e\"" Sep 11 19:46:56.957890 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:56.957809 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4b032e-dockercfg-b2qrb\"" Sep 11 19:46:56.969049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:56.968969 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r"] Sep 11 19:46:57.073293 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.073262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bd6m\" (UniqueName: \"kubernetes.io/projected/f471704e-af44-400e-bb99-cdff6e6fbea7-kube-api-access-8bd6m\") pod \"payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r\" (UID: \"f471704e-af44-400e-bb99-cdff6e6fbea7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:57.073485 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.073325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f471704e-af44-400e-bb99-cdff6e6fbea7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r\" (UID: \"f471704e-af44-400e-bb99-cdff6e6fbea7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:57.174947 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.174903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f471704e-af44-400e-bb99-cdff6e6fbea7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r\" (UID: \"f471704e-af44-400e-bb99-cdff6e6fbea7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:57.175141 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.175037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bd6m\" (UniqueName: \"kubernetes.io/projected/f471704e-af44-400e-bb99-cdff6e6fbea7-kube-api-access-8bd6m\") pod \"payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r\" (UID: \"f471704e-af44-400e-bb99-cdff6e6fbea7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:57.175580 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.175561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f471704e-af44-400e-bb99-cdff6e6fbea7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r\" (UID: \"f471704e-af44-400e-bb99-cdff6e6fbea7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:57.186003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.185973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bd6m\" (UniqueName: \"kubernetes.io/projected/f471704e-af44-400e-bb99-cdff6e6fbea7-kube-api-access-8bd6m\") pod \"payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r\" (UID: \"f471704e-af44-400e-bb99-cdff6e6fbea7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:57.228608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.228524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:57.276781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.276748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:57.567657 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.567621 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r"] Sep 11 19:46:57.764145 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.764103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:57.864958 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.864858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:57.921628 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.921592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" event={"ID":"f471704e-af44-400e-bb99-cdff6e6fbea7","Type":"ContainerStarted","Data":"786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00"} Sep 11 19:46:57.921808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.921690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" event={"ID":"f471704e-af44-400e-bb99-cdff6e6fbea7","Type":"ContainerStarted","Data":"312b1247b66ec6f0d0c9cd71a0b62c350748e327f19434b4bf10187002c5cc76"} Sep 11 19:46:57.921808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.921720 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:46:57.950577 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:57.950506 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" podStartSLOduration=1.950484598 podStartE2EDuration="1.950484598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:57.946798808 +0000 UTC m=+1051.080111394" watchObservedRunningTime="2026-09-11 19:46:57.950484598 +0000 UTC m=+1051.083797193" Sep 11 19:46:58.039852 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:58.038501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:58.132211 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:58.132103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:58.347632 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:58.347599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:58.688811 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:58.688740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:46:59.857245 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:46:59.857217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:00.580425 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:00.580383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:00.776751 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:00.776698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:01.009045 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:01.008943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:01.196750 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:01.196716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:01.351135 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:01.351101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:02.720392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:02.720352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:03.574651 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:03.574611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:03.893300 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:03.893201 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 19:47:03.893300 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:03.893297 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 19:47:04.031017 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:04.030915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:04.132261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:04.132211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:04.518279 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:04.518246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:04.844909 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:04.844874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:04.897019 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:04.896979 2579 patch_prober.go:28] interesting pod/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 19:47:04.897498 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:04.897052 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 19:47:04.898271 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:04.898250 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:47:05.131111 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:05.131022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:05.234780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:05.234705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:05.391103 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:05.391015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:05.953288 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:05.953253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:06.067922 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.067877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:06.171309 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.171254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:06.172318 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.172300 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4"] Sep 11 19:47:06.176315 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.176298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:06.179089 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.179062 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7df2ac\"" Sep 11 19:47:06.179222 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.179111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7df2ac-dockercfg-t777s\"" Sep 11 19:47:06.187147 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.187120 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4"] Sep 11 19:47:06.288536 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.288440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnpc\" (UniqueName: \"kubernetes.io/projected/73b2703a-469f-4250-ba0f-0e6d5c59f126-kube-api-access-rvnpc\") pod \"payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4\" (UID: \"73b2703a-469f-4250-ba0f-0e6d5c59f126\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:06.288536 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.288498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b2703a-469f-4250-ba0f-0e6d5c59f126-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4\" (UID: \"73b2703a-469f-4250-ba0f-0e6d5c59f126\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:06.390236 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.390198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnpc\" (UniqueName: \"kubernetes.io/projected/73b2703a-469f-4250-ba0f-0e6d5c59f126-kube-api-access-rvnpc\") pod \"payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4\" (UID: \"73b2703a-469f-4250-ba0f-0e6d5c59f126\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:06.390236 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.390239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b2703a-469f-4250-ba0f-0e6d5c59f126-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4\" (UID: \"73b2703a-469f-4250-ba0f-0e6d5c59f126\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:06.391041 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.391022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b2703a-469f-4250-ba0f-0e6d5c59f126-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4\" (UID: \"73b2703a-469f-4250-ba0f-0e6d5c59f126\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:06.399878 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.399840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnpc\" (UniqueName: \"kubernetes.io/projected/73b2703a-469f-4250-ba0f-0e6d5c59f126-kube-api-access-rvnpc\") pod \"payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4\" (UID: \"73b2703a-469f-4250-ba0f-0e6d5c59f126\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:06.490247 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.490198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:06.524991 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.524956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:06.628774 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.628656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:06.732502 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.732400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:06.790917 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.790890 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4"] Sep 11 19:47:06.794608 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:47:06.794572 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b2703a_469f_4250_ba0f_0e6d5c59f126.slice/crio-174db83540d1bd0c6b05b09d0defac63b45fffc1ec118dee899f16c494c231b8 WatchSource:0}: Error finding container 174db83540d1bd0c6b05b09d0defac63b45fffc1ec118dee899f16c494c231b8: Status 404 returned error can't find the container with id 174db83540d1bd0c6b05b09d0defac63b45fffc1ec118dee899f16c494c231b8 Sep 11 19:47:06.870716 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.870685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:06.982933 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.982893 2579 generic.go:356] "Generic (PLEG): container finished" podID="29576270-8f07-4992-b746-550d5d05b2f5" containerID="8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4" exitCode=0 Sep 11 19:47:06.983383 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.982936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" event={"ID":"29576270-8f07-4992-b746-550d5d05b2f5","Type":"ContainerDied","Data":"8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4"} Sep 11 19:47:06.983486 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.983467 2579 scope.go:117] "RemoveContainer" containerID="8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4" Sep 11 19:47:06.985048 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.985008 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:47:06.985184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.985094 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:47:06.985184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.985136 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:47:06.985617 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.985594 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" event={"ID":"73b2703a-469f-4250-ba0f-0e6d5c59f126","Type":"ContainerStarted","Data":"d326cba74a10672f37a80d1243445f3edf3f1892cb1f524269d41bfb96c02db0"} Sep 11 19:47:06.985747 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.985642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" event={"ID":"73b2703a-469f-4250-ba0f-0e6d5c59f126","Type":"ContainerStarted","Data":"174db83540d1bd0c6b05b09d0defac63b45fffc1ec118dee899f16c494c231b8"} Sep 11 19:47:06.985747 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:06.985665 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:07.054779 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.054747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:07.106797 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.106687 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" podStartSLOduration=1.106666211 podStartE2EDuration="1.106666211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:07.105523411 +0000 UTC m=+1060.238836001" watchObservedRunningTime="2026-09-11 19:47:07.106666211 +0000 UTC m=+1060.239978803" Sep 11 19:47:07.198157 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.198109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:07.297925 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.297896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:07.517409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.517378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:07.621697 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.621665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:07.671510 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.671414 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g"] Sep 11 19:47:07.674092 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.674067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:07.677507 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.677487 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6fddae31\"" Sep 11 19:47:07.691805 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.691775 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g"] Sep 11 19:47:07.805653 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.805606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee69318-63a0-4ba3-a876-1c684dde0e5d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g\" (UID: \"eee69318-63a0-4ba3-a876-1c684dde0e5d\") " pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:07.805902 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.805789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvdm\" (UniqueName: \"kubernetes.io/projected/eee69318-63a0-4ba3-a876-1c684dde0e5d-kube-api-access-hbvdm\") pod \"payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g\" (UID: \"eee69318-63a0-4ba3-a876-1c684dde0e5d\") " pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:07.906446 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.906404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvdm\" (UniqueName: \"kubernetes.io/projected/eee69318-63a0-4ba3-a876-1c684dde0e5d-kube-api-access-hbvdm\") pod \"payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g\" (UID: \"eee69318-63a0-4ba3-a876-1c684dde0e5d\") " pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:07.906638 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.906496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee69318-63a0-4ba3-a876-1c684dde0e5d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g\" (UID: \"eee69318-63a0-4ba3-a876-1c684dde0e5d\") " pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:07.907286 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.907266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee69318-63a0-4ba3-a876-1c684dde0e5d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g\" (UID: \"eee69318-63a0-4ba3-a876-1c684dde0e5d\") " pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:07.917535 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.917505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvdm\" (UniqueName: \"kubernetes.io/projected/eee69318-63a0-4ba3-a876-1c684dde0e5d-kube-api-access-hbvdm\") pod \"payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g\" (UID: \"eee69318-63a0-4ba3-a876-1c684dde0e5d\") " pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:07.985727 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.985625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:07.994331 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.994306 2579 generic.go:356] "Generic (PLEG): container finished" podID="686e089b-fcff-45f8-8481-67d970a09e8e" containerID="462851e3465c164a16247a7e9611da5e8a944eeeb4cc10406b481c2cf63c2385" exitCode=0 Sep 11 19:47:07.994483 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.994390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" event={"ID":"686e089b-fcff-45f8-8481-67d970a09e8e","Type":"ContainerDied","Data":"462851e3465c164a16247a7e9611da5e8a944eeeb4cc10406b481c2cf63c2385"} Sep 11 19:47:07.994943 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.994926 2579 scope.go:117] "RemoveContainer" containerID="462851e3465c164a16247a7e9611da5e8a944eeeb4cc10406b481c2cf63c2385" Sep 11 19:47:07.996372 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.996340 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:47:07.996455 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.996431 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:47:07.996498 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.996474 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:47:07.996717 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:07.996698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" event={"ID":"29576270-8f07-4992-b746-550d5d05b2f5","Type":"ContainerStarted","Data":"e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1"} Sep 11 19:47:08.203852 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:08.203767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:08.281908 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:08.281238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g"] Sep 11 19:47:08.302040 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:47:08.302000 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee69318_63a0_4ba3_a876_1c684dde0e5d.slice/crio-ee022823a08399ca0c509fd77440a6cf39bea19fc12e3cac2eb1e83bf0d1adaf WatchSource:0}: Error finding container ee022823a08399ca0c509fd77440a6cf39bea19fc12e3cac2eb1e83bf0d1adaf: Status 404 returned error can't find the container with id ee022823a08399ca0c509fd77440a6cf39bea19fc12e3cac2eb1e83bf0d1adaf Sep 11 19:47:08.329846 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:08.329163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:08.698872 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:08.698764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:47:08.699091 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:08.699040 2579 patch_prober.go:28] interesting pod/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 11 19:47:08.699171 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:08.699089 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 11 19:47:08.932570 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:08.931001 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 19:47:09.011909 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.009727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" event={"ID":"eee69318-63a0-4ba3-a876-1c684dde0e5d","Type":"ContainerStarted","Data":"628a2d6184fdf9b0b57b8ec173334c5c317af4a7b1b7074b977af5b902b0427c"} Sep 11 19:47:09.011909 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.009796 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" event={"ID":"eee69318-63a0-4ba3-a876-1c684dde0e5d","Type":"ContainerStarted","Data":"ee022823a08399ca0c509fd77440a6cf39bea19fc12e3cac2eb1e83bf0d1adaf"} Sep 11 19:47:09.011909 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.010669 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:09.017921 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.013983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" event={"ID":"686e089b-fcff-45f8-8481-67d970a09e8e","Type":"ContainerStarted","Data":"3f20fb590d5ddb6de2533f915e17c360404a2f2e67544d47b0c40682fbc38207"} Sep 11 19:47:09.055419 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.055366 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" podStartSLOduration=2.055347129 podStartE2EDuration="2.055347129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:09.05303645 +0000 UTC m=+1062.186349060" watchObservedRunningTime="2026-09-11 19:47:09.055347129 +0000 UTC m=+1062.188659754" Sep 11 19:47:09.413482 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.413437 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:47:09.413674 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.413580 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 11 19:47:09.413674 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.413626 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 11 19:47:09.517315 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.517273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:09.687683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.687568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:09.699491 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.699445 2579 patch_prober.go:28] interesting pod/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 11 19:47:09.699702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:09.699530 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 11 19:47:10.278653 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:10.278587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:10.414319 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:10.414276 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 11 19:47:10.414517 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:10.414359 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 11 19:47:10.698769 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:10.698720 2579 patch_prober.go:28] interesting pod/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 11 19:47:10.699016 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:10.698798 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 11 19:47:11.271104 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:11.271029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:11.413817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:11.413766 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 11 19:47:11.414323 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:11.413863 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 11 19:47:11.560321 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:11.560258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:11.699208 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:11.699169 2579 patch_prober.go:28] interesting pod/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 11 19:47:11.699527 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:11.699496 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 11 19:47:12.003783 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:12.003672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:12.413385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:12.413347 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 11 19:47:12.413622 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:12.413404 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 11 19:47:12.708475 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:12.708228 2579 patch_prober.go:28] interesting pod/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 11 19:47:12.708475 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:12.708291 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 11 19:47:12.736590 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:12.736561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:12.834748 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:12.834713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:13.238561 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.238523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:13.331617 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.331583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:13.397011 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.396974 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64"] Sep 11 19:47:13.399608 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.399586 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:13.402256 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.402233 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-04331c\"" Sep 11 19:47:13.402393 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.402262 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-04331c-dockercfg-76g4j\"" Sep 11 19:47:13.407500 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.407477 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64"] Sep 11 19:47:13.414171 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.414133 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 11 19:47:13.414302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.414189 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 11 19:47:13.429764 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.429723 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2"] Sep 11 19:47:13.438650 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.437700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:13.447765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.447733 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2"] Sep 11 19:47:13.507171 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.507066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:13.523806 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.523764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-04331c-f7d479cf-42r64\" (UID: \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:13.524008 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.523968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psq75\" (UniqueName: \"kubernetes.io/projected/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-kube-api-access-psq75\") pod \"payload-processing-e2e-inf-b-04331c-f7d479cf-42r64\" (UID: \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:13.625135 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.625037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-04331c-f7d479cf-42r64\" (UID: \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:13.625135 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.625130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2befbe77-3c01-4336-aaef-1de2199aa0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2\" (UID: \"2befbe77-3c01-4336-aaef-1de2199aa0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:13.625392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.625173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwpn\" (UniqueName: \"kubernetes.io/projected/2befbe77-3c01-4336-aaef-1de2199aa0ad-kube-api-access-qqwpn\") pod \"payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2\" (UID: \"2befbe77-3c01-4336-aaef-1de2199aa0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:13.625392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.625225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psq75\" (UniqueName: \"kubernetes.io/projected/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-kube-api-access-psq75\") pod \"payload-processing-e2e-inf-b-04331c-f7d479cf-42r64\" (UID: \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:13.625970 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.625890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-04331c-f7d479cf-42r64\" (UID: \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:13.636291 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.636228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psq75\" (UniqueName: \"kubernetes.io/projected/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-kube-api-access-psq75\") pod \"payload-processing-e2e-inf-b-04331c-f7d479cf-42r64\" (UID: \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:13.702367 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.702336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:47:13.702666 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.702640 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:47:13.703367 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.703351 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:47:13.714921 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.714888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:13.726144 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.726119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2befbe77-3c01-4336-aaef-1de2199aa0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2\" (UID: \"2befbe77-3c01-4336-aaef-1de2199aa0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:13.726290 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.726182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwpn\" (UniqueName: \"kubernetes.io/projected/2befbe77-3c01-4336-aaef-1de2199aa0ad-kube-api-access-qqwpn\") pod \"payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2\" (UID: \"2befbe77-3c01-4336-aaef-1de2199aa0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:13.726868 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.726812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2befbe77-3c01-4336-aaef-1de2199aa0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2\" (UID: \"2befbe77-3c01-4336-aaef-1de2199aa0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:13.750080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.750038 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwpn\" (UniqueName: \"kubernetes.io/projected/2befbe77-3c01-4336-aaef-1de2199aa0ad-kube-api-access-qqwpn\") pod \"payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2\" (UID: \"2befbe77-3c01-4336-aaef-1de2199aa0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:13.755670 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.755590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:13.758753 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:13.758606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:14.036044 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:14.036015 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64"] Sep 11 19:47:14.039652 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:47:14.039617 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac433d43_277e_4665_9b32_a0fc9fc3a2cb.slice/crio-e35c24a226042a170352f2b6f7982332fc8899ea7311633c2dfc0c4cd0d68dee WatchSource:0}: Error finding container e35c24a226042a170352f2b6f7982332fc8899ea7311633c2dfc0c4cd0d68dee: Status 404 returned error can't find the container with id e35c24a226042a170352f2b6f7982332fc8899ea7311633c2dfc0c4cd0d68dee Sep 11 19:47:14.122785 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:14.122759 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2"] Sep 11 19:47:14.125077 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:47:14.124999 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2befbe77_3c01_4336_aaef_1de2199aa0ad.slice/crio-046303435ae79d2a63561d594e15c2a81cec19053a7777c58306278080536a9d WatchSource:0}: Error finding container 046303435ae79d2a63561d594e15c2a81cec19053a7777c58306278080536a9d: Status 404 returned error can't find the container with id 046303435ae79d2a63561d594e15c2a81cec19053a7777c58306278080536a9d Sep 11 19:47:14.502958 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:14.502856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:47:14.503490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:14.503459 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:47:14.505087 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:14.505058 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 19:47:14.610672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:14.610637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:15.049396 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:15.049360 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" event={"ID":"ac433d43-277e-4665-9b32-a0fc9fc3a2cb","Type":"ContainerStarted","Data":"6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585"} Sep 11 19:47:15.049900 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:15.049426 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:15.049900 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:15.049453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" event={"ID":"ac433d43-277e-4665-9b32-a0fc9fc3a2cb","Type":"ContainerStarted","Data":"e35c24a226042a170352f2b6f7982332fc8899ea7311633c2dfc0c4cd0d68dee"} Sep 11 19:47:15.050986 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:15.050963 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" event={"ID":"2befbe77-3c01-4336-aaef-1de2199aa0ad","Type":"ContainerStarted","Data":"010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d"} Sep 11 19:47:15.051073 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:15.051023 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:15.051073 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:15.051043 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" event={"ID":"2befbe77-3c01-4336-aaef-1de2199aa0ad","Type":"ContainerStarted","Data":"046303435ae79d2a63561d594e15c2a81cec19053a7777c58306278080536a9d"} Sep 11 19:47:15.069842 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:15.069776 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" podStartSLOduration=2.069762342 podStartE2EDuration="2.069762342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:15.068616743 +0000 UTC m=+1068.201929340" watchObservedRunningTime="2026-09-11 19:47:15.069762342 +0000 UTC m=+1068.203074931" Sep 11 19:47:15.625174 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:15.625144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:16.595775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:16.595744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:17.413041 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:17.413003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:17.977031 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:17.976995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:17.998910 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:17.998874 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:47:18.031197 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:18.031139 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" podStartSLOduration=5.03112295 podStartE2EDuration="5.03112295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:15.088297932 +0000 UTC m=+1068.221610521" watchObservedRunningTime="2026-09-11 19:47:18.03112295 +0000 UTC m=+1071.164435538" Sep 11 19:47:18.075650 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:18.075618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:18.174404 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:18.174376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:18.586883 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:18.586772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:21.028434 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:21.028406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:24.036380 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:24.036346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:24.137722 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:24.137687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:24.237707 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:24.237676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:24.376649 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:24.376564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:26.059019 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:26.058988 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:47:26.059482 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:26.059072 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:47:26.437998 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:26.437870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:26.538988 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:26.538945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:28.106801 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:28.106702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:29.066365 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:29.066322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:30.078983 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:30.078927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:31.138808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:31.138772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:31.249086 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:31.249049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:32.008903 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:32.008868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:32.522080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:32.522048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:32.632219 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:32.632178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:32.740738 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:32.740700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:33.064156 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:33.064103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:33.688691 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:33.688636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:34.688752 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:34.688722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:35.472962 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:35.472934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:36.187505 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.187470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:36.200724 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.200688 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl"] Sep 11 19:47:36.204029 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.204007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" Sep 11 19:47:36.206783 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.206760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:47:36.206923 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.206781 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:47:36.213761 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.213729 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl"] Sep 11 19:47:36.352342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.352310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e4503-a408-4b7c-bfc4-eff116b18bc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6fddae31-h5stl\" (UID: \"b88e4503-a408-4b7c-bfc4-eff116b18bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" Sep 11 19:47:36.453310 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.453225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e4503-a408-4b7c-bfc4-eff116b18bc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6fddae31-h5stl\" (UID: \"b88e4503-a408-4b7c-bfc4-eff116b18bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" Sep 11 19:47:36.453851 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.453814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e4503-a408-4b7c-bfc4-eff116b18bc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6fddae31-h5stl\" (UID: \"b88e4503-a408-4b7c-bfc4-eff116b18bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" Sep 11 19:47:36.515667 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.515630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" Sep 11 19:47:36.801316 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:47:36.801285 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88e4503_a408_4b7c_bfc4_eff116b18bc6.slice/crio-8e999fe1cdbf6ca1ea41ffe767118c40996d914bad544a10ab6fe0a3c8c953fc WatchSource:0}: Error finding container 8e999fe1cdbf6ca1ea41ffe767118c40996d914bad544a10ab6fe0a3c8c953fc: Status 404 returned error can't find the container with id 8e999fe1cdbf6ca1ea41ffe767118c40996d914bad544a10ab6fe0a3c8c953fc Sep 11 19:47:36.802210 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.802179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl"] Sep 11 19:47:36.803794 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:36.803778 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:47:37.188441 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:37.188353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" event={"ID":"b88e4503-a408-4b7c-bfc4-eff116b18bc6","Type":"ContainerStarted","Data":"8e999fe1cdbf6ca1ea41ffe767118c40996d914bad544a10ab6fe0a3c8c953fc"} Sep 11 19:47:37.221057 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:37.221021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:38.057625 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:38.057588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:39.142706 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:39.142671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:40.066655 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:40.066611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:40.207036 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:40.207001 2579 generic.go:356] "Generic (PLEG): container finished" podID="b88e4503-a408-4b7c-bfc4-eff116b18bc6" containerID="cbbdd5b989bf0a1e8a8e849fd734e95c72bdd75041257487151db7e373e79cee" exitCode=0 Sep 11 19:47:40.207430 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:40.207065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" event={"ID":"b88e4503-a408-4b7c-bfc4-eff116b18bc6","Type":"ContainerDied","Data":"cbbdd5b989bf0a1e8a8e849fd734e95c72bdd75041257487151db7e373e79cee"} Sep 11 19:47:40.973437 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:40.973398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:41.354319 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:41.354297 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" Sep 11 19:47:41.507209 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:41.507164 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e4503-a408-4b7c-bfc4-eff116b18bc6-maas-api-service-ca\") pod \"b88e4503-a408-4b7c-bfc4-eff116b18bc6\" (UID: \"b88e4503-a408-4b7c-bfc4-eff116b18bc6\") " Sep 11 19:47:41.507539 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:41.507512 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88e4503-a408-4b7c-bfc4-eff116b18bc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b88e4503-a408-4b7c-bfc4-eff116b18bc6" (UID: "b88e4503-a408-4b7c-bfc4-eff116b18bc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:47:41.507623 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:41.507612 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e4503-a408-4b7c-bfc4-eff116b18bc6-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:47:41.806129 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:41.806101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:42.220284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:42.220199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" event={"ID":"b88e4503-a408-4b7c-bfc4-eff116b18bc6","Type":"ContainerDied","Data":"8e999fe1cdbf6ca1ea41ffe767118c40996d914bad544a10ab6fe0a3c8c953fc"} Sep 11 19:47:42.220284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:42.220247 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e999fe1cdbf6ca1ea41ffe767118c40996d914bad544a10ab6fe0a3c8c953fc" Sep 11 19:47:42.220284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:42.220209 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl" Sep 11 19:47:42.727558 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:42.727523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:46.377549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:46.377514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:46.970554 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:46.970515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:47.003885 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.003849 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g"] Sep 11 19:47:47.004113 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.004085 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" podUID="eee69318-63a0-4ba3-a876-1c684dde0e5d" containerName="payload-processing" containerID="cri-o://628a2d6184fdf9b0b57b8ec173334c5c317af4a7b1b7074b977af5b902b0427c" gracePeriod=30 Sep 11 19:47:47.062105 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.062073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:47.164361 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.164278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:47.215329 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.215265 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl"] Sep 11 19:47:47.219188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.219158 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6fddae31-h5stl"] Sep 11 19:47:47.250775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.250681 2579 generic.go:356] "Generic (PLEG): container finished" podID="eee69318-63a0-4ba3-a876-1c684dde0e5d" containerID="628a2d6184fdf9b0b57b8ec173334c5c317af4a7b1b7074b977af5b902b0427c" exitCode=0 Sep 11 19:47:47.250775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.250751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" event={"ID":"eee69318-63a0-4ba3-a876-1c684dde0e5d","Type":"ContainerDied","Data":"628a2d6184fdf9b0b57b8ec173334c5c317af4a7b1b7074b977af5b902b0427c"} Sep 11 19:47:47.259898 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.259864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:47.305178 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.305153 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:47.396538 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.396501 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b88e4503-a408-4b7c-bfc4-eff116b18bc6" path="/var/lib/kubelet/pods/b88e4503-a408-4b7c-bfc4-eff116b18bc6/volumes" Sep 11 19:47:47.471914 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.471882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbvdm\" (UniqueName: \"kubernetes.io/projected/eee69318-63a0-4ba3-a876-1c684dde0e5d-kube-api-access-hbvdm\") pod \"eee69318-63a0-4ba3-a876-1c684dde0e5d\" (UID: \"eee69318-63a0-4ba3-a876-1c684dde0e5d\") " Sep 11 19:47:47.472246 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.472225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee69318-63a0-4ba3-a876-1c684dde0e5d-plugins-config-volume\") pod \"eee69318-63a0-4ba3-a876-1c684dde0e5d\" (UID: \"eee69318-63a0-4ba3-a876-1c684dde0e5d\") " Sep 11 19:47:47.472875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.472706 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee69318-63a0-4ba3-a876-1c684dde0e5d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eee69318-63a0-4ba3-a876-1c684dde0e5d" (UID: "eee69318-63a0-4ba3-a876-1c684dde0e5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:47:47.473406 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.473384 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee69318-63a0-4ba3-a876-1c684dde0e5d-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:47:47.475290 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.475260 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee69318-63a0-4ba3-a876-1c684dde0e5d-kube-api-access-hbvdm" (OuterVolumeSpecName: "kube-api-access-hbvdm") pod "eee69318-63a0-4ba3-a876-1c684dde0e5d" (UID: "eee69318-63a0-4ba3-a876-1c684dde0e5d"). InnerVolumeSpecName "kube-api-access-hbvdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:47:47.516111 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.516008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:47.574740 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.574696 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbvdm\" (UniqueName: \"kubernetes.io/projected/eee69318-63a0-4ba3-a876-1c684dde0e5d-kube-api-access-hbvdm\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:47:47.688064 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.688013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:47.851024 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.850977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:47.941684 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:47.941650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:48.041324 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.041291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:48.257935 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.257816 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" event={"ID":"eee69318-63a0-4ba3-a876-1c684dde0e5d","Type":"ContainerDied","Data":"ee022823a08399ca0c509fd77440a6cf39bea19fc12e3cac2eb1e83bf0d1adaf"} Sep 11 19:47:48.257935 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.257855 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g" Sep 11 19:47:48.258189 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.257906 2579 scope.go:117] "RemoveContainer" containerID="628a2d6184fdf9b0b57b8ec173334c5c317af4a7b1b7074b977af5b902b0427c" Sep 11 19:47:48.293372 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.293343 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g"] Sep 11 19:47:48.301898 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.301867 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6fddae31-c758fb8c5-qw77g"] Sep 11 19:47:48.324519 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.324489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:48.533650 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.533580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:48.634062 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:48.634036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:49.394362 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:49.394331 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee69318-63a0-4ba3-a876-1c684dde0e5d" path="/var/lib/kubelet/pods/eee69318-63a0-4ba3-a876-1c684dde0e5d/volumes" Sep 11 19:47:49.554703 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:49.554670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:50.628189 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:50.628152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:51.065614 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:51.065585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:51.173025 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:51.172991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:51.611369 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:51.611334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:52.654817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:52.654779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:53.535218 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:53.535187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:53.642048 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:53.641988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:53.988480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:53.988377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:54.065211 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:54.065180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:54.405138 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:54.405081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:55.194159 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:55.194131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:55.499421 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:55.499307 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:55.962891 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:55.962842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:47:56.782603 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:47:56.782562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:15.475035 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.475005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:15.870363 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.870323 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj"] Sep 11 19:48:15.870762 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.870749 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee69318-63a0-4ba3-a876-1c684dde0e5d" containerName="payload-processing" Sep 11 19:48:15.870808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.870764 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee69318-63a0-4ba3-a876-1c684dde0e5d" containerName="payload-processing" Sep 11 19:48:15.870808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.870782 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88e4503-a408-4b7c-bfc4-eff116b18bc6" containerName="revoke-keys" Sep 11 19:48:15.870808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.870787 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88e4503-a408-4b7c-bfc4-eff116b18bc6" containerName="revoke-keys" Sep 11 19:48:15.870926 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.870891 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88e4503-a408-4b7c-bfc4-eff116b18bc6" containerName="revoke-keys" Sep 11 19:48:15.870926 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.870901 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee69318-63a0-4ba3-a876-1c684dde0e5d" containerName="payload-processing" Sep 11 19:48:15.878599 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.878569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" Sep 11 19:48:15.881185 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.881158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj"] Sep 11 19:48:15.881307 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.881200 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:48:15.881307 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.881168 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:48:15.971005 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:15.970965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c97a4-8ee9-4daf-964a-e0945a2d73a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj\" (UID: \"103c97a4-8ee9-4daf-964a-e0945a2d73a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" Sep 11 19:48:16.072216 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:16.072183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c97a4-8ee9-4daf-964a-e0945a2d73a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj\" (UID: \"103c97a4-8ee9-4daf-964a-e0945a2d73a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" Sep 11 19:48:16.072896 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:16.072875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c97a4-8ee9-4daf-964a-e0945a2d73a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj\" (UID: \"103c97a4-8ee9-4daf-964a-e0945a2d73a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" Sep 11 19:48:16.192554 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:16.192461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" Sep 11 19:48:16.374006 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:16.373969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:16.467736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:16.467703 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj"] Sep 11 19:48:16.470475 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:48:16.470436 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103c97a4_8ee9_4daf_964a_e0945a2d73a2.slice/crio-ba1523e2092dc54ce0e2334d692da075b1f8c212e48c87ccdc5f4f0663ed1cfd WatchSource:0}: Error finding container ba1523e2092dc54ce0e2334d692da075b1f8c212e48c87ccdc5f4f0663ed1cfd: Status 404 returned error can't find the container with id ba1523e2092dc54ce0e2334d692da075b1f8c212e48c87ccdc5f4f0663ed1cfd Sep 11 19:48:17.004951 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:17.004923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:17.434692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:17.434598 2579 generic.go:356] "Generic (PLEG): container finished" podID="103c97a4-8ee9-4daf-964a-e0945a2d73a2" containerID="3608eeb35917d14a86ef4b56fabe46cf7981194e6f49d9e92a9c06ea02bac61f" exitCode=0 Sep 11 19:48:17.434857 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:17.434687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" event={"ID":"103c97a4-8ee9-4daf-964a-e0945a2d73a2","Type":"ContainerDied","Data":"3608eeb35917d14a86ef4b56fabe46cf7981194e6f49d9e92a9c06ea02bac61f"} Sep 11 19:48:17.434857 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:17.434771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" event={"ID":"103c97a4-8ee9-4daf-964a-e0945a2d73a2","Type":"ContainerStarted","Data":"ba1523e2092dc54ce0e2334d692da075b1f8c212e48c87ccdc5f4f0663ed1cfd"} Sep 11 19:48:17.788396 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:17.788360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:18.415353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:18.415315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:18.591906 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:18.591876 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" Sep 11 19:48:18.699999 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:18.699908 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c97a4-8ee9-4daf-964a-e0945a2d73a2-maas-api-service-ca\") pod \"103c97a4-8ee9-4daf-964a-e0945a2d73a2\" (UID: \"103c97a4-8ee9-4daf-964a-e0945a2d73a2\") " Sep 11 19:48:18.700276 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:18.700253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103c97a4-8ee9-4daf-964a-e0945a2d73a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "103c97a4-8ee9-4daf-964a-e0945a2d73a2" (UID: "103c97a4-8ee9-4daf-964a-e0945a2d73a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:48:18.801480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:18.801440 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103c97a4-8ee9-4daf-964a-e0945a2d73a2-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:48:19.181420 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:19.181390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:19.447954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:19.447864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" event={"ID":"103c97a4-8ee9-4daf-964a-e0945a2d73a2","Type":"ContainerDied","Data":"ba1523e2092dc54ce0e2334d692da075b1f8c212e48c87ccdc5f4f0663ed1cfd"} Sep 11 19:48:19.447954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:19.447882 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj" Sep 11 19:48:19.447954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:19.447914 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba1523e2092dc54ce0e2334d692da075b1f8c212e48c87ccdc5f4f0663ed1cfd" Sep 11 19:48:20.000309 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:20.000271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:20.752106 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:20.752069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:22.277261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:22.277225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:22.832084 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:22.832045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:22.904753 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:22.904716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:22.967616 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:22.967577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:23.020021 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:23.019981 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj"] Sep 11 19:48:23.023092 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:23.023067 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8fc710a-zjcsj"] Sep 11 19:48:23.035581 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:23.035556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:23.170038 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:23.169950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:23.394476 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:23.394444 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103c97a4-8ee9-4daf-964a-e0945a2d73a2" path="/var/lib/kubelet/pods/103c97a4-8ee9-4daf-964a-e0945a2d73a2/volumes" Sep 11 19:48:23.480494 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:23.480417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:23.601726 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:23.601696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:23.921875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:23.921804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:24.249557 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:24.249431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:24.343303 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:24.343095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:24.977095 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:24.977051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:25.745251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:25.745216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:25.989129 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:25.989094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:26.109217 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:26.109187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:26.581717 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:26.581690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:27.294610 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:27.294581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:27.917174 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:27.917142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:27.987889 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:27.987851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:28.159639 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:28.159609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:28.380518 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:28.380478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:28.480265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:28.480229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:28.769565 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:28.769477 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:29.521363 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:29.521323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:29.665408 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:29.665378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:29.766368 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:29.766335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:30.166010 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:30.165974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:30.884178 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:30.884133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:37.648458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:37.648427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:49.972267 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:49.972187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:50.685803 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:50.685772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:51.380292 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.380255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:51.714957 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.714877 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h"] Sep 11 19:48:51.715389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.715368 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103c97a4-8ee9-4daf-964a-e0945a2d73a2" containerName="revoke-keys" Sep 11 19:48:51.715389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.715385 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="103c97a4-8ee9-4daf-964a-e0945a2d73a2" containerName="revoke-keys" Sep 11 19:48:51.715492 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.715482 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="103c97a4-8ee9-4daf-964a-e0945a2d73a2" containerName="revoke-keys" Sep 11 19:48:51.720009 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.719989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" Sep 11 19:48:51.723139 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.723120 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:48:51.723349 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.723329 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:48:51.737742 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.737712 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h"] Sep 11 19:48:51.843968 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.843927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3d941-0403-483d-b3c1-6f5defdaec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h\" (UID: \"e7c3d941-0403-483d-b3c1-6f5defdaec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" Sep 11 19:48:51.945522 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.945486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3d941-0403-483d-b3c1-6f5defdaec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h\" (UID: \"e7c3d941-0403-483d-b3c1-6f5defdaec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" Sep 11 19:48:51.946170 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:51.946148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3d941-0403-483d-b3c1-6f5defdaec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h\" (UID: \"e7c3d941-0403-483d-b3c1-6f5defdaec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" Sep 11 19:48:52.030711 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:52.030620 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" Sep 11 19:48:52.161552 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:52.161520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:52.303570 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:52.303541 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h"] Sep 11 19:48:52.305437 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:48:52.305402 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c3d941_0403_483d_b3c1_6f5defdaec77.slice/crio-73628a36b20b5387886aee08351c40dd99fbeebee6d81508949d8d544991dce8 WatchSource:0}: Error finding container 73628a36b20b5387886aee08351c40dd99fbeebee6d81508949d8d544991dce8: Status 404 returned error can't find the container with id 73628a36b20b5387886aee08351c40dd99fbeebee6d81508949d8d544991dce8 Sep 11 19:48:52.635281 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:52.635250 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" event={"ID":"e7c3d941-0403-483d-b3c1-6f5defdaec77","Type":"ContainerStarted","Data":"91b506f6cc3928a79de049adc2699adfad61651717c008d857307647d5917432"} Sep 11 19:48:52.635774 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:52.635302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" event={"ID":"e7c3d941-0403-483d-b3c1-6f5defdaec77","Type":"ContainerStarted","Data":"73628a36b20b5387886aee08351c40dd99fbeebee6d81508949d8d544991dce8"} Sep 11 19:48:52.650464 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:52.650406 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" podStartSLOduration=1.650384137 podStartE2EDuration="1.650384137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:48:52.650140896 +0000 UTC m=+1165.783453500" watchObservedRunningTime="2026-09-11 19:48:52.650384137 +0000 UTC m=+1165.783696730" Sep 11 19:48:53.080209 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:53.080176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:53.641990 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:53.641955 2579 generic.go:356] "Generic (PLEG): container finished" podID="e7c3d941-0403-483d-b3c1-6f5defdaec77" containerID="91b506f6cc3928a79de049adc2699adfad61651717c008d857307647d5917432" exitCode=0 Sep 11 19:48:53.642375 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:53.642030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" event={"ID":"e7c3d941-0403-483d-b3c1-6f5defdaec77","Type":"ContainerDied","Data":"91b506f6cc3928a79de049adc2699adfad61651717c008d857307647d5917432"} Sep 11 19:48:53.688639 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:53.688609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:54.316745 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:54.316710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:54.793704 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:54.793680 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" Sep 11 19:48:54.874188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:54.874152 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3d941-0403-483d-b3c1-6f5defdaec77-maas-api-service-ca\") pod \"e7c3d941-0403-483d-b3c1-6f5defdaec77\" (UID: \"e7c3d941-0403-483d-b3c1-6f5defdaec77\") " Sep 11 19:48:54.874526 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:54.874503 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c3d941-0403-483d-b3c1-6f5defdaec77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7c3d941-0403-483d-b3c1-6f5defdaec77" (UID: "e7c3d941-0403-483d-b3c1-6f5defdaec77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:48:54.975989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:54.975905 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3d941-0403-483d-b3c1-6f5defdaec77-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:48:55.069865 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:55.069807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:55.654845 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:55.654733 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" event={"ID":"e7c3d941-0403-483d-b3c1-6f5defdaec77","Type":"ContainerDied","Data":"73628a36b20b5387886aee08351c40dd99fbeebee6d81508949d8d544991dce8"} Sep 11 19:48:55.654845 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:55.654756 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h" Sep 11 19:48:55.654845 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:55.654793 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73628a36b20b5387886aee08351c40dd99fbeebee6d81508949d8d544991dce8" Sep 11 19:48:55.784889 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:55.784849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:56.948130 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:56.948090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:57.214050 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:57.213976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:57.283074 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:57.283046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:57.346892 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:57.346863 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:57.404272 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:57.404241 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h"] Sep 11 19:48:57.406333 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:57.406308 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d9a4dfe-dct7h"] Sep 11 19:48:57.409270 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:57.409244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:57.857677 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:57.857629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:58.507294 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:58.507243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:58.609089 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:58.609036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:58.946755 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:58.946722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:59.350455 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:59.350422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:59.395018 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:59.394985 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c3d941-0403-483d-b3c1-6f5defdaec77" path="/var/lib/kubelet/pods/e7c3d941-0403-483d-b3c1-6f5defdaec77/volumes" Sep 11 19:48:59.454078 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:59.454048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:48:59.551259 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:48:59.551224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:00.181623 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:00.181593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:00.872819 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:00.872787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:01.651541 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:01.651504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:02.190207 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:02.190164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:02.276393 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:02.276357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:02.488992 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:02.488905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:03.112623 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:03.112592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:03.934310 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:03.934278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:04.638662 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:04.638632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:05.268547 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:05.268511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:06.019817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.019786 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:06.137584 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.137551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:06.251574 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.251544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:06.482660 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.482631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:06.585906 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.585867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:06.587490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.587471 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz"] Sep 11 19:49:06.587934 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.587922 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c3d941-0403-483d-b3c1-6f5defdaec77" containerName="revoke-keys" Sep 11 19:49:06.587975 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.587937 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c3d941-0403-483d-b3c1-6f5defdaec77" containerName="revoke-keys" Sep 11 19:49:06.588021 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.588012 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c3d941-0403-483d-b3c1-6f5defdaec77" containerName="revoke-keys" Sep 11 19:49:06.591351 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.591334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:06.594090 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.594070 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6008a04d-dockercfg-ckj8q\"" Sep 11 19:49:06.594374 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.594358 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6008a04d\"" Sep 11 19:49:06.601439 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.601404 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz"] Sep 11 19:49:06.601767 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.601737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5frk6\" (UniqueName: \"kubernetes.io/projected/d9ae3cac-b213-4d87-bce2-dad129179953-kube-api-access-5frk6\") pod \"payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz\" (UID: \"d9ae3cac-b213-4d87-bce2-dad129179953\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:06.601931 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.601850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ae3cac-b213-4d87-bce2-dad129179953-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz\" (UID: \"d9ae3cac-b213-4d87-bce2-dad129179953\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:06.702480 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.702439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ae3cac-b213-4d87-bce2-dad129179953-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz\" (UID: \"d9ae3cac-b213-4d87-bce2-dad129179953\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:06.702668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.702578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5frk6\" (UniqueName: \"kubernetes.io/projected/d9ae3cac-b213-4d87-bce2-dad129179953-kube-api-access-5frk6\") pod \"payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz\" (UID: \"d9ae3cac-b213-4d87-bce2-dad129179953\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:06.703915 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.703892 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ae3cac-b213-4d87-bce2-dad129179953-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz\" (UID: \"d9ae3cac-b213-4d87-bce2-dad129179953\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:06.717856 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.717791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5frk6\" (UniqueName: \"kubernetes.io/projected/d9ae3cac-b213-4d87-bce2-dad129179953-kube-api-access-5frk6\") pod \"payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz\" (UID: \"d9ae3cac-b213-4d87-bce2-dad129179953\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:06.904061 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:06.904005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:07.016839 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:07.016797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:07.183967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:07.183940 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz"] Sep 11 19:49:07.186509 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:49:07.186482 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ae3cac_b213_4d87_bce2_dad129179953.slice/crio-32086cce4163793bb570f641fc4220bc72e4778dbcb33fb731044ea7b715a8bb WatchSource:0}: Error finding container 32086cce4163793bb570f641fc4220bc72e4778dbcb33fb731044ea7b715a8bb: Status 404 returned error can't find the container with id 32086cce4163793bb570f641fc4220bc72e4778dbcb33fb731044ea7b715a8bb Sep 11 19:49:07.727019 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:07.726983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" event={"ID":"d9ae3cac-b213-4d87-bce2-dad129179953","Type":"ContainerStarted","Data":"aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64"} Sep 11 19:49:07.727423 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:07.727044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" event={"ID":"d9ae3cac-b213-4d87-bce2-dad129179953","Type":"ContainerStarted","Data":"32086cce4163793bb570f641fc4220bc72e4778dbcb33fb731044ea7b715a8bb"} Sep 11 19:49:07.727423 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:07.727087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:07.749951 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:07.749892 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" podStartSLOduration=1.749876159 podStartE2EDuration="1.749876159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:49:07.747713257 +0000 UTC m=+1180.881025857" watchObservedRunningTime="2026-09-11 19:49:07.749876159 +0000 UTC m=+1180.883188748" Sep 11 19:49:07.918025 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:07.917994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:11.851426 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:11.851372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:18.735086 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:18.735038 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:27.401138 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:27.401107 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:49:27.403597 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:27.403579 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:49:27.870498 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:27.870461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:28.462369 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:28.462339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:29.285008 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:29.284946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:29.923757 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:29.923720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:30.269281 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:30.269175 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:30.659982 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:30.659954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:31.269747 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:31.269689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:32.136578 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.136548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:32.278026 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.277989 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw"] Sep 11 19:49:32.281866 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.281815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" Sep 11 19:49:32.284980 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.284957 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:49:32.285254 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.285237 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:49:32.290714 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.290496 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw"] Sep 11 19:49:32.365079 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.365039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c015ade8-62e7-4bde-b28e-19600c93be41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw\" (UID: \"c015ade8-62e7-4bde-b28e-19600c93be41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" Sep 11 19:49:32.466765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.466670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c015ade8-62e7-4bde-b28e-19600c93be41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw\" (UID: \"c015ade8-62e7-4bde-b28e-19600c93be41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" Sep 11 19:49:32.467316 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.467293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c015ade8-62e7-4bde-b28e-19600c93be41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw\" (UID: \"c015ade8-62e7-4bde-b28e-19600c93be41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" Sep 11 19:49:32.595546 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.595512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" Sep 11 19:49:32.869215 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:32.869174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw"] Sep 11 19:49:32.870849 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:49:32.870801 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc015ade8_62e7_4bde_b28e_19600c93be41.slice/crio-3784a26953464522d36f6b71c24999566263e715de0a8f26033222e89bec6a61 WatchSource:0}: Error finding container 3784a26953464522d36f6b71c24999566263e715de0a8f26033222e89bec6a61: Status 404 returned error can't find the container with id 3784a26953464522d36f6b71c24999566263e715de0a8f26033222e89bec6a61 Sep 11 19:49:33.091082 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:33.091044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:33.703433 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:33.703388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:33.870921 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:33.870884 2579 generic.go:356] "Generic (PLEG): container finished" podID="c015ade8-62e7-4bde-b28e-19600c93be41" containerID="4ad44a9cac90804e7883204983e75f41b19cba060125c7c0e259761b6b58fafe" exitCode=0 Sep 11 19:49:33.871131 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:33.870970 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" event={"ID":"c015ade8-62e7-4bde-b28e-19600c93be41","Type":"ContainerDied","Data":"4ad44a9cac90804e7883204983e75f41b19cba060125c7c0e259761b6b58fafe"} Sep 11 19:49:33.871131 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:33.871017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" event={"ID":"c015ade8-62e7-4bde-b28e-19600c93be41","Type":"ContainerStarted","Data":"3784a26953464522d36f6b71c24999566263e715de0a8f26033222e89bec6a61"} Sep 11 19:49:34.347885 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:34.347819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:35.030278 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.030250 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" Sep 11 19:49:35.177768 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.177653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:35.194953 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.194927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c015ade8-62e7-4bde-b28e-19600c93be41-maas-api-service-ca\") pod \"c015ade8-62e7-4bde-b28e-19600c93be41\" (UID: \"c015ade8-62e7-4bde-b28e-19600c93be41\") " Sep 11 19:49:35.195349 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.195324 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c015ade8-62e7-4bde-b28e-19600c93be41-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c015ade8-62e7-4bde-b28e-19600c93be41" (UID: "c015ade8-62e7-4bde-b28e-19600c93be41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:49:35.296299 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.296255 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c015ade8-62e7-4bde-b28e-19600c93be41-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:49:35.769552 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.769513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:35.869293 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.869261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:35.884160 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.884122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" event={"ID":"c015ade8-62e7-4bde-b28e-19600c93be41","Type":"ContainerDied","Data":"3784a26953464522d36f6b71c24999566263e715de0a8f26033222e89bec6a61"} Sep 11 19:49:35.884160 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.884173 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw" Sep 11 19:49:35.884382 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.884178 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3784a26953464522d36f6b71c24999566263e715de0a8f26033222e89bec6a61" Sep 11 19:49:35.971076 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:35.971030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:36.453169 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:36.453138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:37.061876 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:37.061805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:43.940091 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:43.940039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:43.970247 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:43.970207 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz"] Sep 11 19:49:43.970503 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:43.970476 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" podUID="d9ae3cac-b213-4d87-bce2-dad129179953" containerName="payload-pre-processing" containerID="cri-o://aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64" gracePeriod=30 Sep 11 19:49:44.041715 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.041688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:44.104892 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.104858 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw"] Sep 11 19:49:44.106706 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.106667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:44.107346 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.107326 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6008a04d-dzqnw"] Sep 11 19:49:44.166302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.166270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:44.266200 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.266111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:44.277733 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.277708 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:44.288479 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.288456 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ae3cac-b213-4d87-bce2-dad129179953-plugins-config-volume\") pod \"d9ae3cac-b213-4d87-bce2-dad129179953\" (UID: \"d9ae3cac-b213-4d87-bce2-dad129179953\") " Sep 11 19:49:44.288584 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.288557 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5frk6\" (UniqueName: \"kubernetes.io/projected/d9ae3cac-b213-4d87-bce2-dad129179953-kube-api-access-5frk6\") pod \"d9ae3cac-b213-4d87-bce2-dad129179953\" (UID: \"d9ae3cac-b213-4d87-bce2-dad129179953\") " Sep 11 19:49:44.288796 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.288775 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ae3cac-b213-4d87-bce2-dad129179953-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9ae3cac-b213-4d87-bce2-dad129179953" (UID: "d9ae3cac-b213-4d87-bce2-dad129179953"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:49:44.290752 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.290729 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ae3cac-b213-4d87-bce2-dad129179953-kube-api-access-5frk6" (OuterVolumeSpecName: "kube-api-access-5frk6") pod "d9ae3cac-b213-4d87-bce2-dad129179953" (UID: "d9ae3cac-b213-4d87-bce2-dad129179953"). InnerVolumeSpecName "kube-api-access-5frk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:49:44.389722 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.389684 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ae3cac-b213-4d87-bce2-dad129179953-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:49:44.389722 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.389710 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5frk6\" (UniqueName: \"kubernetes.io/projected/d9ae3cac-b213-4d87-bce2-dad129179953-kube-api-access-5frk6\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:49:44.570234 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.570198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:44.926006 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.925926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:44.936508 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.936472 2579 generic.go:356] "Generic (PLEG): container finished" podID="d9ae3cac-b213-4d87-bce2-dad129179953" containerID="aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64" exitCode=0 Sep 11 19:49:44.936718 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.936531 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" Sep 11 19:49:44.936718 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.936537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" event={"ID":"d9ae3cac-b213-4d87-bce2-dad129179953","Type":"ContainerDied","Data":"aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64"} Sep 11 19:49:44.936718 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.936644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz" event={"ID":"d9ae3cac-b213-4d87-bce2-dad129179953","Type":"ContainerDied","Data":"32086cce4163793bb570f641fc4220bc72e4778dbcb33fb731044ea7b715a8bb"} Sep 11 19:49:44.936718 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.936667 2579 scope.go:117] "RemoveContainer" containerID="aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64" Sep 11 19:49:44.946570 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.946549 2579 scope.go:117] "RemoveContainer" containerID="aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64" Sep 11 19:49:44.946912 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:49:44.946889 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64\": container with ID starting with aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64 not found: ID does not exist" containerID="aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64" Sep 11 19:49:44.946965 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.946922 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64"} err="failed to get container status \"aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64\": rpc error: code = NotFound desc = could not find container \"aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64\": container with ID starting with aca132e9e6e123d26ae80950471478acd98d3f626e51a107024e34dd735a5a64 not found: ID does not exist" Sep 11 19:49:44.958819 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.958779 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz"] Sep 11 19:49:44.962737 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:44.962710 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6008a04d-7b6f6f8c54-8xdxz"] Sep 11 19:49:45.024034 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.024006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:45.067741 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.067697 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l"] Sep 11 19:49:45.068276 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.068255 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c015ade8-62e7-4bde-b28e-19600c93be41" containerName="revoke-keys" Sep 11 19:49:45.068376 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.068277 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c015ade8-62e7-4bde-b28e-19600c93be41" containerName="revoke-keys" Sep 11 19:49:45.068376 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.068293 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ae3cac-b213-4d87-bce2-dad129179953" containerName="payload-pre-processing" Sep 11 19:49:45.068376 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.068301 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ae3cac-b213-4d87-bce2-dad129179953" containerName="payload-pre-processing" Sep 11 19:49:45.068535 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.068423 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ae3cac-b213-4d87-bce2-dad129179953" containerName="payload-pre-processing" Sep 11 19:49:45.068535 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.068435 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c015ade8-62e7-4bde-b28e-19600c93be41" containerName="revoke-keys" Sep 11 19:49:45.073050 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.073026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" Sep 11 19:49:45.075464 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.075444 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:49:45.075572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.075445 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:49:45.079210 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.079190 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l"] Sep 11 19:49:45.097240 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.097210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4622c401-b5a1-4ed6-9410-05f11d6d6d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7df2ac-7g82l\" (UID: \"4622c401-b5a1-4ed6-9410-05f11d6d6d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" Sep 11 19:49:45.098213 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.098183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:45.198389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.198303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4622c401-b5a1-4ed6-9410-05f11d6d6d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7df2ac-7g82l\" (UID: \"4622c401-b5a1-4ed6-9410-05f11d6d6d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" Sep 11 19:49:45.198920 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.198902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4622c401-b5a1-4ed6-9410-05f11d6d6d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7df2ac-7g82l\" (UID: \"4622c401-b5a1-4ed6-9410-05f11d6d6d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" Sep 11 19:49:45.354383 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.354355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:45.386309 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.386272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" Sep 11 19:49:45.394352 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.394317 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c015ade8-62e7-4bde-b28e-19600c93be41" path="/var/lib/kubelet/pods/c015ade8-62e7-4bde-b28e-19600c93be41/volumes" Sep 11 19:49:45.394883 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.394859 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ae3cac-b213-4d87-bce2-dad129179953" path="/var/lib/kubelet/pods/d9ae3cac-b213-4d87-bce2-dad129179953/volumes" Sep 11 19:49:45.668055 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.668028 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l"] Sep 11 19:49:45.669619 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:49:45.669588 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4622c401_b5a1_4ed6_9410_05f11d6d6d4a.slice/crio-58c9e80ea1ad1a925debb34c161c217e11037db161760cd639a0f8b1137db8be WatchSource:0}: Error finding container 58c9e80ea1ad1a925debb34c161c217e11037db161760cd639a0f8b1137db8be: Status 404 returned error can't find the container with id 58c9e80ea1ad1a925debb34c161c217e11037db161760cd639a0f8b1137db8be Sep 11 19:49:45.942958 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:45.942865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" event={"ID":"4622c401-b5a1-4ed6-9410-05f11d6d6d4a","Type":"ContainerStarted","Data":"58c9e80ea1ad1a925debb34c161c217e11037db161760cd639a0f8b1137db8be"} Sep 11 19:49:46.074355 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:46.074327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:46.370052 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:46.370023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:46.685167 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:46.685085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:46.951856 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:46.951740 2579 generic.go:356] "Generic (PLEG): container finished" podID="4622c401-b5a1-4ed6-9410-05f11d6d6d4a" containerID="1a1bb1e418f22dd729f2b78ae5c37cbf84baf5bd59fe31877a49df7da0e2b702" exitCode=0 Sep 11 19:49:46.951856 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:46.951816 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" event={"ID":"4622c401-b5a1-4ed6-9410-05f11d6d6d4a","Type":"ContainerDied","Data":"1a1bb1e418f22dd729f2b78ae5c37cbf84baf5bd59fe31877a49df7da0e2b702"} Sep 11 19:49:47.475538 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:47.475502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:48.093806 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.093773 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" Sep 11 19:49:48.095366 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.095337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:48.125879 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.125852 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4622c401-b5a1-4ed6-9410-05f11d6d6d4a-maas-api-service-ca\") pod \"4622c401-b5a1-4ed6-9410-05f11d6d6d4a\" (UID: \"4622c401-b5a1-4ed6-9410-05f11d6d6d4a\") " Sep 11 19:49:48.126290 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.126261 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4622c401-b5a1-4ed6-9410-05f11d6d6d4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4622c401-b5a1-4ed6-9410-05f11d6d6d4a" (UID: "4622c401-b5a1-4ed6-9410-05f11d6d6d4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:49:48.227402 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.227303 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4622c401-b5a1-4ed6-9410-05f11d6d6d4a-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:49:48.772353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.772324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:48.963595 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.963572 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" Sep 11 19:49:48.963786 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.963564 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l" event={"ID":"4622c401-b5a1-4ed6-9410-05f11d6d6d4a","Type":"ContainerDied","Data":"58c9e80ea1ad1a925debb34c161c217e11037db161760cd639a0f8b1137db8be"} Sep 11 19:49:48.963786 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:48.963700 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c9e80ea1ad1a925debb34c161c217e11037db161760cd639a0f8b1137db8be" Sep 11 19:49:52.153144 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:52.153114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:52.230137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:52.230110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:52.328047 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:52.328016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:52.426740 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:52.426651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:52.831326 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:52.831294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:54.157002 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:54.156971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:55.206052 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:55.206021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:55.780655 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:55.780622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:55.820258 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:55.820224 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4"] Sep 11 19:49:55.820505 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:55.820480 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" podUID="73b2703a-469f-4250-ba0f-0e6d5c59f126" containerName="payload-pre-processing" containerID="cri-o://d326cba74a10672f37a80d1243445f3edf3f1892cb1f524269d41bfb96c02db0" gracePeriod=30 Sep 11 19:49:55.879742 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:55.879714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:55.961141 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:55.961101 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l"] Sep 11 19:49:55.962198 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:55.962165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:55.963661 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:55.963633 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7df2ac-7g82l"] Sep 11 19:49:56.005111 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.005076 2579 generic.go:356] "Generic (PLEG): container finished" podID="73b2703a-469f-4250-ba0f-0e6d5c59f126" containerID="d326cba74a10672f37a80d1243445f3edf3f1892cb1f524269d41bfb96c02db0" exitCode=0 Sep 11 19:49:56.005288 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.005138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" event={"ID":"73b2703a-469f-4250-ba0f-0e6d5c59f126","Type":"ContainerDied","Data":"d326cba74a10672f37a80d1243445f3edf3f1892cb1f524269d41bfb96c02db0"} Sep 11 19:49:56.022245 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.022216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:56.107940 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.107914 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:49:56.212095 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.212056 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvnpc\" (UniqueName: \"kubernetes.io/projected/73b2703a-469f-4250-ba0f-0e6d5c59f126-kube-api-access-rvnpc\") pod \"73b2703a-469f-4250-ba0f-0e6d5c59f126\" (UID: \"73b2703a-469f-4250-ba0f-0e6d5c59f126\") " Sep 11 19:49:56.212495 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.212129 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b2703a-469f-4250-ba0f-0e6d5c59f126-plugins-config-volume\") pod \"73b2703a-469f-4250-ba0f-0e6d5c59f126\" (UID: \"73b2703a-469f-4250-ba0f-0e6d5c59f126\") " Sep 11 19:49:56.212540 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.212484 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b2703a-469f-4250-ba0f-0e6d5c59f126-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73b2703a-469f-4250-ba0f-0e6d5c59f126" (UID: "73b2703a-469f-4250-ba0f-0e6d5c59f126"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:49:56.214676 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.214644 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b2703a-469f-4250-ba0f-0e6d5c59f126-kube-api-access-rvnpc" (OuterVolumeSpecName: "kube-api-access-rvnpc") pod "73b2703a-469f-4250-ba0f-0e6d5c59f126" (UID: "73b2703a-469f-4250-ba0f-0e6d5c59f126"). InnerVolumeSpecName "kube-api-access-rvnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:49:56.313787 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.313744 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvnpc\" (UniqueName: \"kubernetes.io/projected/73b2703a-469f-4250-ba0f-0e6d5c59f126-kube-api-access-rvnpc\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:49:56.313787 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.313777 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b2703a-469f-4250-ba0f-0e6d5c59f126-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:49:56.474246 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.474147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:56.724231 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:56.724201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:57.012792 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.012761 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" event={"ID":"73b2703a-469f-4250-ba0f-0e6d5c59f126","Type":"ContainerDied","Data":"174db83540d1bd0c6b05b09d0defac63b45fffc1ec118dee899f16c494c231b8"} Sep 11 19:49:57.012997 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.012835 2579 scope.go:117] "RemoveContainer" containerID="d326cba74a10672f37a80d1243445f3edf3f1892cb1f524269d41bfb96c02db0" Sep 11 19:49:57.012997 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.012763 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4" Sep 11 19:49:57.034786 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.034755 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4"] Sep 11 19:49:57.038430 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.038405 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7df2ac-7b5d5b6846-8bch4"] Sep 11 19:49:57.098717 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.098685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:57.394696 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.394666 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4622c401-b5a1-4ed6-9410-05f11d6d6d4a" path="/var/lib/kubelet/pods/4622c401-b5a1-4ed6-9410-05f11d6d6d4a/volumes" Sep 11 19:49:57.395086 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.395077 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73b2703a-469f-4250-ba0f-0e6d5c59f126" path="/var/lib/kubelet/pods/73b2703a-469f-4250-ba0f-0e6d5c59f126/volumes" Sep 11 19:49:57.873302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:57.873273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:58.490785 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:58.490746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:49:59.121356 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:49:59.121304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:01.800184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:01.800126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:13.692675 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:13.692612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:18.760171 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:18.760055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:20.982706 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:20.982670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:24.623502 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:24.623470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:24.733657 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:24.733620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:24.825260 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:24.825228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:34.479122 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:34.479087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:38.466080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:38.466046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:39.215271 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:39.215231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:49.840343 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:49.840308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:50.140928 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:50.140809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:50.241409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:50.241370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:50.673141 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:50.673112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:53.149593 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:53.149563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:53.316605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:53.316572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:53.411844 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:53.411744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:54.176729 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:54.176692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:55.012694 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:55.012660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:55.743563 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:55.743519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:56.604965 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:56.604932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:56.706264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:56.706231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:56.835541 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:56.835506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:57.067174 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.067134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:57.170329 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.170282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:57.265226 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.265191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:50:57.299125 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.299084 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74"] Sep 11 19:50:57.299551 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.299538 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4622c401-b5a1-4ed6-9410-05f11d6d6d4a" containerName="revoke-keys" Sep 11 19:50:57.299601 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.299553 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4622c401-b5a1-4ed6-9410-05f11d6d6d4a" containerName="revoke-keys" Sep 11 19:50:57.299601 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.299581 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b2703a-469f-4250-ba0f-0e6d5c59f126" containerName="payload-pre-processing" Sep 11 19:50:57.299601 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.299587 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b2703a-469f-4250-ba0f-0e6d5c59f126" containerName="payload-pre-processing" Sep 11 19:50:57.299693 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.299653 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4622c401-b5a1-4ed6-9410-05f11d6d6d4a" containerName="revoke-keys" Sep 11 19:50:57.299693 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.299661 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b2703a-469f-4250-ba0f-0e6d5c59f126" containerName="payload-pre-processing" Sep 11 19:50:57.302924 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.302907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:57.305965 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.305945 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f76076\"" Sep 11 19:50:57.306059 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.305945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f76076-dockercfg-x8ccm\"" Sep 11 19:50:57.312876 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.312853 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74"] Sep 11 19:50:57.350747 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.350654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwrmq\" (UniqueName: \"kubernetes.io/projected/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-kube-api-access-gwrmq\") pod \"payload-processing-e2e-explicit-f76076-7bf454798f-59j74\" (UID: \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\") " pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:57.350747 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.350738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f76076-7bf454798f-59j74\" (UID: \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\") " pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:57.372750 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.372716 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g"] Sep 11 19:50:57.376362 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.376344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:57.395267 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.395240 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g"] Sep 11 19:50:57.451625 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.451584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f76076-7bf454798f-59j74\" (UID: \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\") " pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:57.451836 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.451666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e8a49-a030-4ba8-a44f-4239c5969b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g\" (UID: \"695e8a49-a030-4ba8-a44f-4239c5969b18\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:57.451836 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.451698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwrmq\" (UniqueName: \"kubernetes.io/projected/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-kube-api-access-gwrmq\") pod \"payload-processing-e2e-explicit-f76076-7bf454798f-59j74\" (UID: \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\") " pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:57.451836 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.451719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wdn\" (UniqueName: \"kubernetes.io/projected/695e8a49-a030-4ba8-a44f-4239c5969b18-kube-api-access-p9wdn\") pod \"payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g\" (UID: \"695e8a49-a030-4ba8-a44f-4239c5969b18\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:57.452333 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.452314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f76076-7bf454798f-59j74\" (UID: \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\") " pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:57.464101 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.464076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwrmq\" (UniqueName: \"kubernetes.io/projected/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-kube-api-access-gwrmq\") pod \"payload-processing-e2e-explicit-f76076-7bf454798f-59j74\" (UID: \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\") " pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:57.553421 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.553380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e8a49-a030-4ba8-a44f-4239c5969b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g\" (UID: \"695e8a49-a030-4ba8-a44f-4239c5969b18\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:57.553592 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.553440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wdn\" (UniqueName: \"kubernetes.io/projected/695e8a49-a030-4ba8-a44f-4239c5969b18-kube-api-access-p9wdn\") pod \"payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g\" (UID: \"695e8a49-a030-4ba8-a44f-4239c5969b18\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:57.554126 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.554105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e8a49-a030-4ba8-a44f-4239c5969b18-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g\" (UID: \"695e8a49-a030-4ba8-a44f-4239c5969b18\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:57.573174 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.573139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wdn\" (UniqueName: \"kubernetes.io/projected/695e8a49-a030-4ba8-a44f-4239c5969b18-kube-api-access-p9wdn\") pod \"payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g\" (UID: \"695e8a49-a030-4ba8-a44f-4239c5969b18\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:57.614793 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.614698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:57.686314 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.686048 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:57.891848 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.891808 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74"] Sep 11 19:50:57.893577 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:50:57.893555 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e16ddc7_ddd4_4166_85b2_1b8c4a726b83.slice/crio-e5e924a9415e9f0e39c9d996384323ed4ffd46fe08b696a78a86fc4cd5e10d72 WatchSource:0}: Error finding container e5e924a9415e9f0e39c9d996384323ed4ffd46fe08b696a78a86fc4cd5e10d72: Status 404 returned error can't find the container with id e5e924a9415e9f0e39c9d996384323ed4ffd46fe08b696a78a86fc4cd5e10d72 Sep 11 19:50:57.973041 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:57.973016 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g"] Sep 11 19:50:57.975536 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:50:57.975505 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695e8a49_a030_4ba8_a44f_4239c5969b18.slice/crio-0e2af414f8f02cf93ab16cb78d6ca5b74d17d9cf21bf0d5d1794f83c811bf405 WatchSource:0}: Error finding container 0e2af414f8f02cf93ab16cb78d6ca5b74d17d9cf21bf0d5d1794f83c811bf405: Status 404 returned error can't find the container with id 0e2af414f8f02cf93ab16cb78d6ca5b74d17d9cf21bf0d5d1794f83c811bf405 Sep 11 19:50:58.370017 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:58.369970 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" event={"ID":"695e8a49-a030-4ba8-a44f-4239c5969b18","Type":"ContainerStarted","Data":"c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a"} Sep 11 19:50:58.370244 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:58.370046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" event={"ID":"695e8a49-a030-4ba8-a44f-4239c5969b18","Type":"ContainerStarted","Data":"0e2af414f8f02cf93ab16cb78d6ca5b74d17d9cf21bf0d5d1794f83c811bf405"} Sep 11 19:50:58.370244 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:58.370081 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:50:58.371883 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:58.371859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" event={"ID":"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83","Type":"ContainerStarted","Data":"1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383"} Sep 11 19:50:58.371989 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:58.371906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" event={"ID":"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83","Type":"ContainerStarted","Data":"e5e924a9415e9f0e39c9d996384323ed4ffd46fe08b696a78a86fc4cd5e10d72"} Sep 11 19:50:58.372067 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:58.372052 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:50:58.387313 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:58.387261 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" podStartSLOduration=1.3872437899999999 podStartE2EDuration="1.38724379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:50:58.386060142 +0000 UTC m=+1291.519372733" watchObservedRunningTime="2026-09-11 19:50:58.38724379 +0000 UTC m=+1291.520556415" Sep 11 19:50:58.405317 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:50:58.405265 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" podStartSLOduration=1.405250712 podStartE2EDuration="1.405250712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:50:58.403912187 +0000 UTC m=+1291.537224777" watchObservedRunningTime="2026-09-11 19:50:58.405250712 +0000 UTC m=+1291.538563300" Sep 11 19:51:07.758284 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:07.758252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:08.519674 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:08.519620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:09.380395 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:09.380311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:51:09.380753 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:09.380506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:51:14.511627 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:14.511599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:14.600035 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:14.600005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:14.704281 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:14.704252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:14.799431 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:14.799398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:15.439950 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:15.439921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:15.539961 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:15.539922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:15.640241 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:15.640207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:17.152720 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:17.152689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:19.738254 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:19.738217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:20.581981 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:20.581949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:21.277111 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:21.277074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:22.094288 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:22.094250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:22.735279 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:22.735247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:23.316663 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:23.316629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:23.416088 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:23.416053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:23.516180 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:23.516150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:24.057721 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:24.057686 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:29.982149 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:29.982114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:30.079474 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:30.079436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:30.191959 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:30.191906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:30.504167 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:30.504112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:31.270692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:31.270658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:35.696646 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:35.696610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:35.798484 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:35.798450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:36.274209 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:36.274177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:36.984160 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:36.984123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:37.077689 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:37.077662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:37.775878 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:37.775844 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:38.653336 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:38.653303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:39.381318 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:39.381283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:40.043206 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:40.043173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:41.016838 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.016790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:41.150127 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.150090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:41.514961 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.514931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:41.612953 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.612910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:51:41.637445 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.637410 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc"] Sep 11 19:51:41.641428 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.641408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:41.644343 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.644319 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-81969518\"" Sep 11 19:51:41.644478 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.644319 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-81969518-dockercfg-fbqf9\"" Sep 11 19:51:41.657549 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.657520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc"] Sep 11 19:51:41.714133 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.714092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fdf\" (UniqueName: \"kubernetes.io/projected/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-kube-api-access-m5fdf\") pod \"payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc\" (UID: \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:41.714668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.714640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc\" (UID: \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:41.815935 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.815895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc\" (UID: \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:41.816148 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.816012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fdf\" (UniqueName: \"kubernetes.io/projected/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-kube-api-access-m5fdf\") pod \"payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc\" (UID: \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:41.816645 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.816627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc\" (UID: \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:41.826118 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.826089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fdf\" (UniqueName: \"kubernetes.io/projected/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-kube-api-access-m5fdf\") pod \"payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc\" (UID: \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:41.952550 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:41.952503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:42.228353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:42.228319 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc"] Sep 11 19:51:42.230705 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:51:42.230678 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c69a6f_41ca_4cdd_b3c7_649d7166ed1c.slice/crio-7ff94957513674a7641cd601a7555f4dedaef1c85786f83a5b033fb78c0fd766 WatchSource:0}: Error finding container 7ff94957513674a7641cd601a7555f4dedaef1c85786f83a5b033fb78c0fd766: Status 404 returned error can't find the container with id 7ff94957513674a7641cd601a7555f4dedaef1c85786f83a5b033fb78c0fd766 Sep 11 19:51:42.639653 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:42.639618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" event={"ID":"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c","Type":"ContainerStarted","Data":"8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e"} Sep 11 19:51:42.639856 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:42.639698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" event={"ID":"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c","Type":"ContainerStarted","Data":"7ff94957513674a7641cd601a7555f4dedaef1c85786f83a5b033fb78c0fd766"} Sep 11 19:51:42.639856 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:42.639720 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:51:42.660547 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:42.660486 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" podStartSLOduration=1.660465976 podStartE2EDuration="1.660465976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:42.659769543 +0000 UTC m=+1335.793082133" watchObservedRunningTime="2026-09-11 19:51:42.660465976 +0000 UTC m=+1335.793778565" Sep 11 19:51:53.648047 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:51:53.647973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:52:01.031415 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:01.031376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:01.107092 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:01.107060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:02.959417 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:02.959380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:03.715798 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:03.715756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:04.471496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:04.471443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:05.104276 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:05.104244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:05.783997 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:05.783965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:06.562933 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:06.562898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:07.008369 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:07.008263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:07.369301 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:07.369267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:07.631152 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:07.631070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:07.729744 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:07.729717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:08.028155 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:08.028062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:14.135517 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.135485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:14.307473 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.307433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:14.408404 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.408313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:14.638395 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.638363 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47"] Sep 11 19:52:14.642436 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.642418 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" Sep 11 19:52:14.645382 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.645354 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:52:14.645456 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.645406 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:52:14.651409 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.651389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47"] Sep 11 19:52:14.760291 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.760199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be442e15-ee2d-421c-ab5a-c0e48588e164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81969518-52v47\" (UID: \"be442e15-ee2d-421c-ab5a-c0e48588e164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" Sep 11 19:52:14.861371 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.861334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be442e15-ee2d-421c-ab5a-c0e48588e164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81969518-52v47\" (UID: \"be442e15-ee2d-421c-ab5a-c0e48588e164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" Sep 11 19:52:14.862031 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.862012 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be442e15-ee2d-421c-ab5a-c0e48588e164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81969518-52v47\" (UID: \"be442e15-ee2d-421c-ab5a-c0e48588e164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" Sep 11 19:52:14.954790 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:14.954752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" Sep 11 19:52:15.241811 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:15.241786 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47"] Sep 11 19:52:15.244047 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:52:15.244014 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe442e15_ee2d_421c_ab5a_c0e48588e164.slice/crio-f52f54a354c7ee2468136556911b5f3414ebec4d2a609005d608dc6fafc2813b WatchSource:0}: Error finding container f52f54a354c7ee2468136556911b5f3414ebec4d2a609005d608dc6fafc2813b: Status 404 returned error can't find the container with id f52f54a354c7ee2468136556911b5f3414ebec4d2a609005d608dc6fafc2813b Sep 11 19:52:15.588195 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:15.588166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:15.839145 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:15.839050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" event={"ID":"be442e15-ee2d-421c-ab5a-c0e48588e164","Type":"ContainerStarted","Data":"cde2e61ef6bb5264e564222283640463a2a5ad30d05ced241f64607ea55afbe1"} Sep 11 19:52:15.839145 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:15.839108 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" event={"ID":"be442e15-ee2d-421c-ab5a-c0e48588e164","Type":"ContainerStarted","Data":"f52f54a354c7ee2468136556911b5f3414ebec4d2a609005d608dc6fafc2813b"} Sep 11 19:52:15.856486 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:15.856433 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" podStartSLOduration=1.856416675 podStartE2EDuration="1.856416675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:15.854132155 +0000 UTC m=+1368.987444755" watchObservedRunningTime="2026-09-11 19:52:15.856416675 +0000 UTC m=+1368.989729264" Sep 11 19:52:16.267033 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:16.266946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:16.846580 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:16.846545 2579 generic.go:356] "Generic (PLEG): container finished" podID="be442e15-ee2d-421c-ab5a-c0e48588e164" containerID="cde2e61ef6bb5264e564222283640463a2a5ad30d05ced241f64607ea55afbe1" exitCode=0 Sep 11 19:52:16.846780 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:16.846636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" event={"ID":"be442e15-ee2d-421c-ab5a-c0e48588e164","Type":"ContainerDied","Data":"cde2e61ef6bb5264e564222283640463a2a5ad30d05ced241f64607ea55afbe1"} Sep 11 19:52:17.054877 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:17.054809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:17.729552 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:17.729522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:17.984960 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:17.984896 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" Sep 11 19:52:18.095004 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:18.094970 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be442e15-ee2d-421c-ab5a-c0e48588e164-maas-api-service-ca\") pod \"be442e15-ee2d-421c-ab5a-c0e48588e164\" (UID: \"be442e15-ee2d-421c-ab5a-c0e48588e164\") " Sep 11 19:52:18.095320 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:18.095299 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be442e15-ee2d-421c-ab5a-c0e48588e164-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be442e15-ee2d-421c-ab5a-c0e48588e164" (UID: "be442e15-ee2d-421c-ab5a-c0e48588e164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:52:18.196832 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:18.196790 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be442e15-ee2d-421c-ab5a-c0e48588e164-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:52:18.376243 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:18.376215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:18.862593 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:18.862568 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" Sep 11 19:52:18.863024 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:18.862610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47" event={"ID":"be442e15-ee2d-421c-ab5a-c0e48588e164","Type":"ContainerDied","Data":"f52f54a354c7ee2468136556911b5f3414ebec4d2a609005d608dc6fafc2813b"} Sep 11 19:52:18.863024 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:18.862665 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f52f54a354c7ee2468136556911b5f3414ebec4d2a609005d608dc6fafc2813b" Sep 11 19:52:19.107696 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:19.107669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:19.758881 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:19.758844 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:20.592521 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:20.592487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:21.250470 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:21.250426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:24.765623 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:24.765592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:25.427978 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.427943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:25.457169 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.457107 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc"] Sep 11 19:52:25.457406 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.457381 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" podUID="86c69a6f-41ca-4cdd-b3c7-649d7166ed1c" containerName="payload-pre-processing" containerID="cri-o://8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e" gracePeriod=30 Sep 11 19:52:25.528508 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.528475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:25.611768 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.611724 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47"] Sep 11 19:52:25.613248 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.613222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:25.614235 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.614215 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81969518-52v47"] Sep 11 19:52:25.671360 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.671327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:25.753228 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.753193 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:52:25.774445 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.774409 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5fdf\" (UniqueName: \"kubernetes.io/projected/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-kube-api-access-m5fdf\") pod \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\" (UID: \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\") " Sep 11 19:52:25.774858 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.774565 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-plugins-config-volume\") pod \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\" (UID: \"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c\") " Sep 11 19:52:25.774961 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.774942 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86c69a6f-41ca-4cdd-b3c7-649d7166ed1c" (UID: "86c69a6f-41ca-4cdd-b3c7-649d7166ed1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:52:25.777073 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.777049 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-kube-api-access-m5fdf" (OuterVolumeSpecName: "kube-api-access-m5fdf") pod "86c69a6f-41ca-4cdd-b3c7-649d7166ed1c" (UID: "86c69a6f-41ca-4cdd-b3c7-649d7166ed1c"). InnerVolumeSpecName "kube-api-access-m5fdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:52:25.876003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.875954 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:52:25.876003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.875994 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fdf\" (UniqueName: \"kubernetes.io/projected/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c-kube-api-access-m5fdf\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:52:25.908258 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.908207 2579 generic.go:356] "Generic (PLEG): container finished" podID="86c69a6f-41ca-4cdd-b3c7-649d7166ed1c" containerID="8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e" exitCode=0 Sep 11 19:52:25.908427 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.908381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" event={"ID":"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c","Type":"ContainerDied","Data":"8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e"} Sep 11 19:52:25.908507 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.908466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" event={"ID":"86c69a6f-41ca-4cdd-b3c7-649d7166ed1c","Type":"ContainerDied","Data":"7ff94957513674a7641cd601a7555f4dedaef1c85786f83a5b033fb78c0fd766"} Sep 11 19:52:25.908561 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.908505 2579 scope.go:117] "RemoveContainer" containerID="8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e" Sep 11 19:52:25.908878 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.908702 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc" Sep 11 19:52:25.920184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.920164 2579 scope.go:117] "RemoveContainer" containerID="8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e" Sep 11 19:52:25.920484 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:52:25.920463 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e\": container with ID starting with 8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e not found: ID does not exist" containerID="8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e" Sep 11 19:52:25.920559 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.920496 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e"} err="failed to get container status \"8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e\": rpc error: code = NotFound desc = could not find container \"8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e\": container with ID starting with 8f1d0e0f91acdd12d7bcc56c71f1eebe10c28761acd660e9a087e9326d87411e not found: ID does not exist" Sep 11 19:52:25.934399 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.934364 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc"] Sep 11 19:52:25.938540 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.938510 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81969518-6949cd47c4-d8pkc"] Sep 11 19:52:25.955668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:25.955622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:26.085361 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:26.085328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:26.770312 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:26.770279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:27.390863 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:27.390790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:27.394745 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:27.394719 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c69a6f-41ca-4cdd-b3c7-649d7166ed1c" path="/var/lib/kubelet/pods/86c69a6f-41ca-4cdd-b3c7-649d7166ed1c/volumes" Sep 11 19:52:27.395076 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:27.395062 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be442e15-ee2d-421c-ab5a-c0e48588e164" path="/var/lib/kubelet/pods/be442e15-ee2d-421c-ab5a-c0e48588e164/volumes" Sep 11 19:52:28.190107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:28.190078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:28.814458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:28.814425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:29.556525 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:29.556494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:32.165286 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:32.165243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:32.236242 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:32.236181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:32.412159 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:32.412115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:32.741444 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:32.741416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:32.838922 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:32.838891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:32.942530 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:32.942474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:38.671288 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:38.671254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:38.732125 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:38.732087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:38.824687 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:38.824651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:38.980693 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:38.980600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:39.078188 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:39.078159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:44.820500 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:44.820456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:44.901298 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:44.901246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:45.698281 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.698252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:45.962575 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.962485 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn"] Sep 11 19:52:45.963036 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.962965 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c69a6f-41ca-4cdd-b3c7-649d7166ed1c" containerName="payload-pre-processing" Sep 11 19:52:45.963036 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.962980 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c69a6f-41ca-4cdd-b3c7-649d7166ed1c" containerName="payload-pre-processing" Sep 11 19:52:45.963036 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.962995 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be442e15-ee2d-421c-ab5a-c0e48588e164" containerName="revoke-keys" Sep 11 19:52:45.963036 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.963001 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be442e15-ee2d-421c-ab5a-c0e48588e164" containerName="revoke-keys" Sep 11 19:52:45.963237 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.963076 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c69a6f-41ca-4cdd-b3c7-649d7166ed1c" containerName="payload-pre-processing" Sep 11 19:52:45.963237 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.963086 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be442e15-ee2d-421c-ab5a-c0e48588e164" containerName="revoke-keys" Sep 11 19:52:45.967925 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.967899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:45.970936 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.970914 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e8cfcf-openshift-default-dockercfg-jxrj9\"" Sep 11 19:52:45.979448 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.979422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn"] Sep 11 19:52:45.986381 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:45.986357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:46.086462 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.086417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-credential-socket\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.086971 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.086949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-podinfo\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.087283 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.087264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-data\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.087436 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.087421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgk6d\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-kube-api-access-hgk6d\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.087652 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.087637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-envoy\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.087800 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.087786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istiod-ca-cert\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.088805 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.088767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-certs\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.089038 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.089020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-socket\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.089246 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.089227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-token\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-credential-socket\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-podinfo\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-data\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgk6d\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-kube-api-access-hgk6d\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-envoy\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istiod-ca-cert\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-certs\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-socket\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.190972 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.190962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-token\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.191327 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.191117 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-credential-socket\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.191327 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.191175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-data\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.191424 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.191403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-certs\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.191468 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.191439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-socket\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.192010 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.191842 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istiod-ca-cert\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.193645 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.193620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-envoy\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.193873 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.193855 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-podinfo\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.202449 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.202427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-token\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.202564 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.202518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgk6d\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-kube-api-access-hgk6d\") pod \"e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.275783 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.275705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:46.282109 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.282086 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:46.564678 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.564653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn"] Sep 11 19:52:46.567463 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:52:46.567433 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9940a5_44e6_479c_a5bb_c89c837ea9d2.slice/crio-c2482a8a3613236890c830d1153c56f645d62ceb0a12ee5a0007b0415a46936f WatchSource:0}: Error finding container c2482a8a3613236890c830d1153c56f645d62ceb0a12ee5a0007b0415a46936f: Status 404 returned error can't find the container with id c2482a8a3613236890c830d1153c56f645d62ceb0a12ee5a0007b0415a46936f Sep 11 19:52:46.569663 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.569631 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:52:46.569724 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.569692 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:52:46.569724 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:46.569721 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:52:47.032440 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:47.032339 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" event={"ID":"5f9940a5-44e6-479c-a5bb-c89c837ea9d2","Type":"ContainerStarted","Data":"276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5"} Sep 11 19:52:47.032440 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:47.032404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" event={"ID":"5f9940a5-44e6-479c-a5bb-c89c837ea9d2","Type":"ContainerStarted","Data":"c2482a8a3613236890c830d1153c56f645d62ceb0a12ee5a0007b0415a46936f"} Sep 11 19:52:47.061593 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:47.061538 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" podStartSLOduration=2.061519461 podStartE2EDuration="2.061519461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:47.060023544 +0000 UTC m=+1400.193336134" watchObservedRunningTime="2026-09-11 19:52:47.061519461 +0000 UTC m=+1400.194832061" Sep 11 19:52:47.282952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:47.282865 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:47.287672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:47.287642 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:48.038117 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:48.038085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:48.039068 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:48.039046 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:52:48.194574 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:48.194543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:48.714190 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:48.714149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:48.984018 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:48.983941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:49.079899 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:49.079866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:49.774963 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:49.774930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:50.549979 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:50.549945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:51.183355 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:51.183323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:51.824765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:51.824724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:52.576278 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:52.576247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:52.678955 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:52.678923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:52.952963 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:52.952869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:53.055679 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.055652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:53.138334 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.138281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:52:53.139808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.139736 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c"] Sep 11 19:52:53.143771 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.143751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:53.146974 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.146946 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e8cfcf-dockercfg-4b49b\"" Sep 11 19:52:53.146974 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.146946 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e8cfcf\"" Sep 11 19:52:53.155680 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.155410 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c"] Sep 11 19:52:53.262067 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.261948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ace00-7762-46ca-b910-5ab54c59b690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c\" (UID: \"6b3ace00-7762-46ca-b910-5ab54c59b690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:53.262513 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.262466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpkm7\" (UniqueName: \"kubernetes.io/projected/6b3ace00-7762-46ca-b910-5ab54c59b690-kube-api-access-gpkm7\") pod \"payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c\" (UID: \"6b3ace00-7762-46ca-b910-5ab54c59b690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:53.363320 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.363285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpkm7\" (UniqueName: \"kubernetes.io/projected/6b3ace00-7762-46ca-b910-5ab54c59b690-kube-api-access-gpkm7\") pod \"payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c\" (UID: \"6b3ace00-7762-46ca-b910-5ab54c59b690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:53.363517 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.363380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ace00-7762-46ca-b910-5ab54c59b690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c\" (UID: \"6b3ace00-7762-46ca-b910-5ab54c59b690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:53.364194 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.364174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ace00-7762-46ca-b910-5ab54c59b690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c\" (UID: \"6b3ace00-7762-46ca-b910-5ab54c59b690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:53.371513 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.371483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpkm7\" (UniqueName: \"kubernetes.io/projected/6b3ace00-7762-46ca-b910-5ab54c59b690-kube-api-access-gpkm7\") pod \"payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c\" (UID: \"6b3ace00-7762-46ca-b910-5ab54c59b690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:53.462305 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.462269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:53.735428 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:53.735399 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c"] Sep 11 19:52:53.737364 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:52:53.737332 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3ace00_7762_46ca_b910_5ab54c59b690.slice/crio-dda5f6a84277d320269a0f9874957c2e497dae4a6eb1ffbefbd7a33a34ca99be WatchSource:0}: Error finding container dda5f6a84277d320269a0f9874957c2e497dae4a6eb1ffbefbd7a33a34ca99be: Status 404 returned error can't find the container with id dda5f6a84277d320269a0f9874957c2e497dae4a6eb1ffbefbd7a33a34ca99be Sep 11 19:52:54.075322 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:54.075288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" event={"ID":"6b3ace00-7762-46ca-b910-5ab54c59b690","Type":"ContainerStarted","Data":"174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939"} Sep 11 19:52:54.075702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:54.075342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" event={"ID":"6b3ace00-7762-46ca-b910-5ab54c59b690","Type":"ContainerStarted","Data":"dda5f6a84277d320269a0f9874957c2e497dae4a6eb1ffbefbd7a33a34ca99be"} Sep 11 19:52:54.075702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:54.075367 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:52:54.097789 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:54.097727 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" podStartSLOduration=1.097706691 podStartE2EDuration="1.097706691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:54.094946992 +0000 UTC m=+1407.228259581" watchObservedRunningTime="2026-09-11 19:52:54.097706691 +0000 UTC m=+1407.231019289" Sep 11 19:52:57.839113 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:52:57.839077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:05.083320 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:05.083285 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 19:53:14.571009 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:14.570964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:15.352034 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:15.352003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:16.094120 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.094085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:16.215562 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.215534 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq"] Sep 11 19:53:16.219392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.219372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" Sep 11 19:53:16.222145 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.222124 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:53:16.222257 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.222144 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:53:16.227867 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.227812 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq"] Sep 11 19:53:16.297567 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.297521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5d680-a44e-456b-ab65-d5a32623aa5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f76076-49cwq\" (UID: \"1ae5d680-a44e-456b-ab65-d5a32623aa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" Sep 11 19:53:16.398662 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.398566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5d680-a44e-456b-ab65-d5a32623aa5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f76076-49cwq\" (UID: \"1ae5d680-a44e-456b-ab65-d5a32623aa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" Sep 11 19:53:16.399205 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.399188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5d680-a44e-456b-ab65-d5a32623aa5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f76076-49cwq\" (UID: \"1ae5d680-a44e-456b-ab65-d5a32623aa5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" Sep 11 19:53:16.531321 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.531284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" Sep 11 19:53:16.782910 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.780711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:16.830387 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:16.830306 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq"] Sep 11 19:53:16.833200 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:53:16.833169 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae5d680_a44e_456b_ab65_d5a32623aa5b.slice/crio-f20c2d3ef5fed9484aedbd842b784aa17c64ddcd1d8c5c15252c0fa4a5d88d66 WatchSource:0}: Error finding container f20c2d3ef5fed9484aedbd842b784aa17c64ddcd1d8c5c15252c0fa4a5d88d66: Status 404 returned error can't find the container with id f20c2d3ef5fed9484aedbd842b784aa17c64ddcd1d8c5c15252c0fa4a5d88d66 Sep 11 19:53:17.212481 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:17.212445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" event={"ID":"1ae5d680-a44e-456b-ab65-d5a32623aa5b","Type":"ContainerStarted","Data":"b73437e57a5d04ac0093714bf389544e9ffe3b32bcdabe43c9ad92f713041df3"} Sep 11 19:53:17.212951 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:17.212497 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" event={"ID":"1ae5d680-a44e-456b-ab65-d5a32623aa5b","Type":"ContainerStarted","Data":"f20c2d3ef5fed9484aedbd842b784aa17c64ddcd1d8c5c15252c0fa4a5d88d66"} Sep 11 19:53:17.227543 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:17.227494 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" podStartSLOduration=1.227478306 podStartE2EDuration="1.227478306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:17.227419538 +0000 UTC m=+1430.360732146" watchObservedRunningTime="2026-09-11 19:53:17.227478306 +0000 UTC m=+1430.360790894" Sep 11 19:53:17.668307 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:17.668265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:18.220302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:18.220263 2579 generic.go:356] "Generic (PLEG): container finished" podID="1ae5d680-a44e-456b-ab65-d5a32623aa5b" containerID="b73437e57a5d04ac0093714bf389544e9ffe3b32bcdabe43c9ad92f713041df3" exitCode=0 Sep 11 19:53:18.220693 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:18.220330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" event={"ID":"1ae5d680-a44e-456b-ab65-d5a32623aa5b","Type":"ContainerDied","Data":"b73437e57a5d04ac0093714bf389544e9ffe3b32bcdabe43c9ad92f713041df3"} Sep 11 19:53:18.280136 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:18.280085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:19.430859 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.430796 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" Sep 11 19:53:19.530177 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.530096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5d680-a44e-456b-ab65-d5a32623aa5b-maas-api-service-ca\") pod \"1ae5d680-a44e-456b-ab65-d5a32623aa5b\" (UID: \"1ae5d680-a44e-456b-ab65-d5a32623aa5b\") " Sep 11 19:53:19.530528 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.530502 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae5d680-a44e-456b-ab65-d5a32623aa5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ae5d680-a44e-456b-ab65-d5a32623aa5b" (UID: "1ae5d680-a44e-456b-ab65-d5a32623aa5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:53:19.530598 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.530583 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae5d680-a44e-456b-ab65-d5a32623aa5b-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:53:19.595702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.595664 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4"] Sep 11 19:53:19.596960 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.596938 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae5d680-a44e-456b-ab65-d5a32623aa5b" containerName="revoke-keys" Sep 11 19:53:19.597141 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.597130 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae5d680-a44e-456b-ab65-d5a32623aa5b" containerName="revoke-keys" Sep 11 19:53:19.597394 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.597383 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae5d680-a44e-456b-ab65-d5a32623aa5b" containerName="revoke-keys" Sep 11 19:53:19.603214 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.603190 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4"] Sep 11 19:53:19.603474 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.603460 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.607520 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.607301 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f7fd24-openshift-default-dockercfg-nn5sq\"" Sep 11 19:53:19.732917 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.732875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c206e47f-3c55-4953-ad34-34f1e242579e-istio-podinfo\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.733132 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.732958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-credential-socket\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.733132 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.732999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8mq9\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-kube-api-access-t8mq9\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.733132 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.733021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-socket\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.733132 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.733046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-envoy\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.733424 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.733200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-istio-token\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.733424 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.733261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-certs\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.733424 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.733287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-data\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.733424 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.733323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c206e47f-3c55-4953-ad34-34f1e242579e-istiod-ca-cert\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.754353 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.754320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:19.835041 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c206e47f-3c55-4953-ad34-34f1e242579e-istiod-ca-cert\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.835250 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c206e47f-3c55-4953-ad34-34f1e242579e-istio-podinfo\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.835250 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-credential-socket\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.835250 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mq9\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-kube-api-access-t8mq9\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.835250 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-socket\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.835250 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-envoy\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.835496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-istio-token\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.835496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-certs\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.835496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-data\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.836020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-data\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.836020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-certs\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.836020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c206e47f-3c55-4953-ad34-34f1e242579e-istiod-ca-cert\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.836020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.835978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-socket\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.836020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.836013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-credential-socket\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.837968 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.837948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c206e47f-3c55-4953-ad34-34f1e242579e-istio-podinfo\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.838228 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.838207 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-envoy\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.846178 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.846138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-istio-token\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.846880 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.846817 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8mq9\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-kube-api-access-t8mq9\") pod \"e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.855965 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.855928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:19.918046 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.918010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:19.958062 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:19.958031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:20.065683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.065635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:20.227142 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.227109 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4"] Sep 11 19:53:20.228490 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:53:20.228466 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc206e47f_3c55_4953_ad34_34f1e242579e.slice/crio-b81ea5318883df5f95f97a9188f8a4c37530003c32e5afe30faa5c523fcbd02f WatchSource:0}: Error finding container b81ea5318883df5f95f97a9188f8a4c37530003c32e5afe30faa5c523fcbd02f: Status 404 returned error can't find the container with id b81ea5318883df5f95f97a9188f8a4c37530003c32e5afe30faa5c523fcbd02f Sep 11 19:53:20.231165 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.231133 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:53:20.231238 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.231201 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:53:20.231238 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.231228 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:53:20.234452 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.234432 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" Sep 11 19:53:20.234452 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.234441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq" event={"ID":"1ae5d680-a44e-456b-ab65-d5a32623aa5b","Type":"ContainerDied","Data":"f20c2d3ef5fed9484aedbd842b784aa17c64ddcd1d8c5c15252c0fa4a5d88d66"} Sep 11 19:53:20.234575 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.234483 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f20c2d3ef5fed9484aedbd842b784aa17c64ddcd1d8c5c15252c0fa4a5d88d66" Sep 11 19:53:20.653514 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:20.653483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:21.243018 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:21.242981 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" event={"ID":"c206e47f-3c55-4953-ad34-34f1e242579e","Type":"ContainerStarted","Data":"ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3"} Sep 11 19:53:21.243184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:21.243037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" event={"ID":"c206e47f-3c55-4953-ad34-34f1e242579e","Type":"ContainerStarted","Data":"b81ea5318883df5f95f97a9188f8a4c37530003c32e5afe30faa5c523fcbd02f"} Sep 11 19:53:21.267779 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:21.267725 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" podStartSLOduration=2.267705103 podStartE2EDuration="2.267705103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:21.265420923 +0000 UTC m=+1434.398733523" watchObservedRunningTime="2026-09-11 19:53:21.267705103 +0000 UTC m=+1434.401017692" Sep 11 19:53:21.371210 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:21.371177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:21.648341 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:21.648308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:21.918205 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:21.918115 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:21.924413 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:21.924383 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:22.250762 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:22.250687 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:22.251730 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:22.251708 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:53:22.513644 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:22.513539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:23.151492 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:23.151455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:23.250761 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:23.250721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:23.378406 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:23.378383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:23.477855 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:23.477802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:23.988659 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:23.988627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:24.272406 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:24.272319 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerID="276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5" exitCode=0 Sep 11 19:53:24.272802 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:24.272404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" event={"ID":"5f9940a5-44e6-479c-a5bb-c89c837ea9d2","Type":"ContainerDied","Data":"276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5"} Sep 11 19:53:24.273026 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:24.273002 2579 scope.go:117] "RemoveContainer" containerID="276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5" Sep 11 19:53:24.274203 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:24.274174 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:53:24.274300 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:24.274253 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:53:24.274365 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:24.274304 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:53:24.766004 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:24.765975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:25.280672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:25.280640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" event={"ID":"5f9940a5-44e6-479c-a5bb-c89c837ea9d2","Type":"ContainerStarted","Data":"11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2"} Sep 11 19:53:25.283172 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:25.283154 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:53:25.283347 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:25.283327 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 19:53:25.283388 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:25.283371 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 19:53:25.392456 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:25.392419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:25.974329 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:25.974290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:26.283301 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:26.283196 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 19:53:26.283895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:26.283860 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 19:53:26.597614 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:26.597573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:27.223802 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.223773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:27.284365 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.284314 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 19:53:27.284934 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.284400 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 19:53:27.323050 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.323009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:27.373252 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.373211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:27.670922 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.670879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:27.781354 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.781317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:27.807047 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.807010 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n"] Sep 11 19:53:27.811814 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.811793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:27.816290 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.816263 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f7fd24-dockercfg-xzz2h\"" Sep 11 19:53:27.816440 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.816320 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f7fd24\"" Sep 11 19:53:27.820606 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.820583 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n"] Sep 11 19:53:27.927582 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.927484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgsv\" (UniqueName: \"kubernetes.io/projected/234dd7fd-2d83-47d1-8ffe-f028db4d536b-kube-api-access-hcgsv\") pod \"payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n\" (UID: \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:27.927582 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:27.927556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234dd7fd-2d83-47d1-8ffe-f028db4d536b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n\" (UID: \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:28.028428 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.028380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgsv\" (UniqueName: \"kubernetes.io/projected/234dd7fd-2d83-47d1-8ffe-f028db4d536b-kube-api-access-hcgsv\") pod \"payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n\" (UID: \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:28.028618 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.028446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234dd7fd-2d83-47d1-8ffe-f028db4d536b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n\" (UID: \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:28.029345 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.029321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234dd7fd-2d83-47d1-8ffe-f028db4d536b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n\" (UID: \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:28.037592 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.037564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgsv\" (UniqueName: \"kubernetes.io/projected/234dd7fd-2d83-47d1-8ffe-f028db4d536b-kube-api-access-hcgsv\") pod \"payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n\" (UID: \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:28.068058 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.068017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:28.126165 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.126129 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:28.285760 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.285279 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 19:53:28.285760 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.285347 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 19:53:28.434092 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.434064 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n"] Sep 11 19:53:28.435544 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:53:28.435512 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234dd7fd_2d83_47d1_8ffe_f028db4d536b.slice/crio-b942aa46081f5c93d062ad2234a7a93228c3c34c0f5ee5f2f72232798ed96239 WatchSource:0}: Error finding container b942aa46081f5c93d062ad2234a7a93228c3c34c0f5ee5f2f72232798ed96239: Status 404 returned error can't find the container with id b942aa46081f5c93d062ad2234a7a93228c3c34c0f5ee5f2f72232798ed96239 Sep 11 19:53:28.798930 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:28.798885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:29.283504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.283382 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 19:53:29.283504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.283469 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 19:53:29.318184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.318153 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" event={"ID":"234dd7fd-2d83-47d1-8ffe-f028db4d536b","Type":"ContainerStarted","Data":"fa55afd203a2eb04c1cee5c0cc81157fd5b842f201f243871c129512136aca1b"} Sep 11 19:53:29.318572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.318216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" event={"ID":"234dd7fd-2d83-47d1-8ffe-f028db4d536b","Type":"ContainerStarted","Data":"b942aa46081f5c93d062ad2234a7a93228c3c34c0f5ee5f2f72232798ed96239"} Sep 11 19:53:29.318919 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.318896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:29.346574 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.346489 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" podStartSLOduration=2.3464665780000002 podStartE2EDuration="2.346466578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:29.342594166 +0000 UTC m=+1442.475906754" watchObservedRunningTime="2026-09-11 19:53:29.346466578 +0000 UTC m=+1442.479779164" Sep 11 19:53:29.379045 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.379010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:29.405215 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.405181 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74"] Sep 11 19:53:29.408105 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.407686 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" podUID="6e16ddc7-ddd4-4166-85b2-1b8c4a726b83" containerName="payload-processing" containerID="cri-o://1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383" gracePeriod=30 Sep 11 19:53:29.416422 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.416097 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g"] Sep 11 19:53:29.416422 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.416331 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" podUID="695e8a49-a030-4ba8-a44f-4239c5969b18" containerName="payload-pre-processing" containerID="cri-o://c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a" gracePeriod=30 Sep 11 19:53:29.466077 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.460529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:29.550378 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.550340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:29.629759 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.626440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:29.746993 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.746800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:29.802394 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.802368 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:53:29.819996 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.819892 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq"] Sep 11 19:53:29.828950 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.828200 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f76076-49cwq"] Sep 11 19:53:29.836414 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.836135 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:53:29.850024 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.849988 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwrmq\" (UniqueName: \"kubernetes.io/projected/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-kube-api-access-gwrmq\") pod \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\" (UID: \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\") " Sep 11 19:53:29.850024 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.850022 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-plugins-config-volume\") pod \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\" (UID: \"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83\") " Sep 11 19:53:29.850563 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.850539 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e16ddc7-ddd4-4166-85b2-1b8c4a726b83" (UID: "6e16ddc7-ddd4-4166-85b2-1b8c4a726b83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:53:29.852592 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.852568 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-kube-api-access-gwrmq" (OuterVolumeSpecName: "kube-api-access-gwrmq") pod "6e16ddc7-ddd4-4166-85b2-1b8c4a726b83" (UID: "6e16ddc7-ddd4-4166-85b2-1b8c4a726b83"). InnerVolumeSpecName "kube-api-access-gwrmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:53:29.895346 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.895312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:29.951335 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.951296 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e8a49-a030-4ba8-a44f-4239c5969b18-plugins-config-volume\") pod \"695e8a49-a030-4ba8-a44f-4239c5969b18\" (UID: \"695e8a49-a030-4ba8-a44f-4239c5969b18\") " Sep 11 19:53:29.951534 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.951367 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9wdn\" (UniqueName: \"kubernetes.io/projected/695e8a49-a030-4ba8-a44f-4239c5969b18-kube-api-access-p9wdn\") pod \"695e8a49-a030-4ba8-a44f-4239c5969b18\" (UID: \"695e8a49-a030-4ba8-a44f-4239c5969b18\") " Sep 11 19:53:29.951605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.951596 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwrmq\" (UniqueName: \"kubernetes.io/projected/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-kube-api-access-gwrmq\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:53:29.951669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.951608 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:53:29.951790 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.951759 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695e8a49-a030-4ba8-a44f-4239c5969b18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "695e8a49-a030-4ba8-a44f-4239c5969b18" (UID: "695e8a49-a030-4ba8-a44f-4239c5969b18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:53:29.953571 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:29.953549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695e8a49-a030-4ba8-a44f-4239c5969b18-kube-api-access-p9wdn" (OuterVolumeSpecName: "kube-api-access-p9wdn") pod "695e8a49-a030-4ba8-a44f-4239c5969b18" (UID: "695e8a49-a030-4ba8-a44f-4239c5969b18"). InnerVolumeSpecName "kube-api-access-p9wdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:53:30.053458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.053333 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9wdn\" (UniqueName: \"kubernetes.io/projected/695e8a49-a030-4ba8-a44f-4239c5969b18-kube-api-access-p9wdn\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:53:30.053458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.053375 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e8a49-a030-4ba8-a44f-4239c5969b18-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:53:30.158367 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.158338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:30.283529 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.283481 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 19:53:30.283714 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.283564 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 19:53:30.326074 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.325987 2579 generic.go:356] "Generic (PLEG): container finished" podID="695e8a49-a030-4ba8-a44f-4239c5969b18" containerID="c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a" exitCode=0 Sep 11 19:53:30.326074 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.326035 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" Sep 11 19:53:30.326074 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.326053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" event={"ID":"695e8a49-a030-4ba8-a44f-4239c5969b18","Type":"ContainerDied","Data":"c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a"} Sep 11 19:53:30.326496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.326152 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g" event={"ID":"695e8a49-a030-4ba8-a44f-4239c5969b18","Type":"ContainerDied","Data":"0e2af414f8f02cf93ab16cb78d6ca5b74d17d9cf21bf0d5d1794f83c811bf405"} Sep 11 19:53:30.326496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.326208 2579 scope.go:117] "RemoveContainer" containerID="c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a" Sep 11 19:53:30.327627 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.327607 2579 generic.go:356] "Generic (PLEG): container finished" podID="6e16ddc7-ddd4-4166-85b2-1b8c4a726b83" containerID="1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383" exitCode=0 Sep 11 19:53:30.327708 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.327695 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" Sep 11 19:53:30.327768 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.327734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" event={"ID":"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83","Type":"ContainerDied","Data":"1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383"} Sep 11 19:53:30.327804 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.327791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74" event={"ID":"6e16ddc7-ddd4-4166-85b2-1b8c4a726b83","Type":"ContainerDied","Data":"e5e924a9415e9f0e39c9d996384323ed4ffd46fe08b696a78a86fc4cd5e10d72"} Sep 11 19:53:30.336871 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.336856 2579 scope.go:117] "RemoveContainer" containerID="c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a" Sep 11 19:53:30.337252 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:53:30.337229 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a\": container with ID starting with c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a not found: ID does not exist" containerID="c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a" Sep 11 19:53:30.337327 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.337262 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a"} err="failed to get container status \"c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a\": rpc error: code = NotFound desc = could not find container \"c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a\": container with ID starting with c458e102d526bc0262cb6b19c5f85ff9c9f46c09d3fbd1b46877e20003fff41a not found: ID does not exist" Sep 11 19:53:30.337327 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.337280 2579 scope.go:117] "RemoveContainer" containerID="1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383" Sep 11 19:53:30.345945 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.345928 2579 scope.go:117] "RemoveContainer" containerID="1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383" Sep 11 19:53:30.346242 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:53:30.346223 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383\": container with ID starting with 1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383 not found: ID does not exist" containerID="1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383" Sep 11 19:53:30.346298 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.346249 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383"} err="failed to get container status \"1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383\": rpc error: code = NotFound desc = could not find container \"1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383\": container with ID starting with 1058ca83e2c6148b0a0a0e10151d567f80ed248ecb53c307d14b051df6c42383 not found: ID does not exist" Sep 11 19:53:30.355560 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.355533 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g"] Sep 11 19:53:30.362850 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.361147 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f76076-766bcf977d-9xf6g"] Sep 11 19:53:30.373253 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.373214 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74"] Sep 11 19:53:30.378172 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.378121 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f76076-7bf454798f-59j74"] Sep 11 19:53:30.579841 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:30.579691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:31.287272 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:31.287242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:53:31.287557 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:31.287539 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:53:31.288320 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:31.288300 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 19:53:31.395350 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:31.395322 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae5d680-a44e-456b-ab65-d5a32623aa5b" path="/var/lib/kubelet/pods/1ae5d680-a44e-456b-ab65-d5a32623aa5b/volumes" Sep 11 19:53:31.395737 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:31.395686 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695e8a49-a030-4ba8-a44f-4239c5969b18" path="/var/lib/kubelet/pods/695e8a49-a030-4ba8-a44f-4239c5969b18/volumes" Sep 11 19:53:31.396133 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:31.396109 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e16ddc7-ddd4-4166-85b2-1b8c4a726b83" path="/var/lib/kubelet/pods/6e16ddc7-ddd4-4166-85b2-1b8c4a726b83/volumes" Sep 11 19:53:31.496975 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:31.496935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:32.148952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:32.148922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:32.954592 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:32.954565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:33.614061 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:33.614029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:34.262509 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:34.262474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:34.973373 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:34.973346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:35.229183 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:35.229089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:36.188036 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:36.188004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:37.003615 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:37.003577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:37.805360 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:37.805325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:37.902740 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:37.902704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:38.677618 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:38.677587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:39.471509 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:39.471479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:40.252302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:40.252250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:41.081316 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:41.081284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:41.338244 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:41.338162 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 19:53:41.556942 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:41.556916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:41.779391 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:41.779311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:42.188559 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.188529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:42.253932 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.253900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:42.352091 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.352052 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx"] Sep 11 19:53:42.352573 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.352558 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e16ddc7-ddd4-4166-85b2-1b8c4a726b83" containerName="payload-processing" Sep 11 19:53:42.352619 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.352576 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e16ddc7-ddd4-4166-85b2-1b8c4a726b83" containerName="payload-processing" Sep 11 19:53:42.352619 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.352603 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695e8a49-a030-4ba8-a44f-4239c5969b18" containerName="payload-pre-processing" Sep 11 19:53:42.352619 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.352609 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="695e8a49-a030-4ba8-a44f-4239c5969b18" containerName="payload-pre-processing" Sep 11 19:53:42.352712 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.352682 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="695e8a49-a030-4ba8-a44f-4239c5969b18" containerName="payload-pre-processing" Sep 11 19:53:42.352712 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.352690 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e16ddc7-ddd4-4166-85b2-1b8c4a726b83" containerName="payload-processing" Sep 11 19:53:42.357254 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.357236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" Sep 11 19:53:42.360621 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.360597 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:53:42.360745 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.360602 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:53:42.363068 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.363046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx"] Sep 11 19:53:42.477736 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.477634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:42.489003 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.488972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c6c11-7caf-48b5-af4b-8471da3a498d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx\" (UID: \"946c6c11-7caf-48b5-af4b-8471da3a498d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" Sep 11 19:53:42.579705 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.579660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:42.590217 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.590186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c6c11-7caf-48b5-af4b-8471da3a498d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx\" (UID: \"946c6c11-7caf-48b5-af4b-8471da3a498d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" Sep 11 19:53:42.590766 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.590747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c6c11-7caf-48b5-af4b-8471da3a498d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx\" (UID: \"946c6c11-7caf-48b5-af4b-8471da3a498d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" Sep 11 19:53:42.669011 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.668976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:42.670984 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.670954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" Sep 11 19:53:42.998166 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:42.998142 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx"] Sep 11 19:53:43.000582 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:53:43.000513 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946c6c11_7caf_48b5_af4b_8471da3a498d.slice/crio-c059dcd2554db0387f582cf7a68b539c6a1dcbf784af938201a4c507f7c3bdf4 WatchSource:0}: Error finding container c059dcd2554db0387f582cf7a68b539c6a1dcbf784af938201a4c507f7c3bdf4: Status 404 returned error can't find the container with id c059dcd2554db0387f582cf7a68b539c6a1dcbf784af938201a4c507f7c3bdf4 Sep 11 19:53:43.372330 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:43.372299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:43.408967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:43.408932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" event={"ID":"946c6c11-7caf-48b5-af4b-8471da3a498d","Type":"ContainerStarted","Data":"21e8214bdbccedff34df269b1a75bb5fe07558264480908152353e505d8ed448"} Sep 11 19:53:43.409138 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:43.408997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" event={"ID":"946c6c11-7caf-48b5-af4b-8471da3a498d","Type":"ContainerStarted","Data":"c059dcd2554db0387f582cf7a68b539c6a1dcbf784af938201a4c507f7c3bdf4"} Sep 11 19:53:43.428058 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:43.428007 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" podStartSLOduration=1.427990571 podStartE2EDuration="1.427990571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:43.4253672 +0000 UTC m=+1456.558679791" watchObservedRunningTime="2026-09-11 19:53:43.427990571 +0000 UTC m=+1456.561303160" Sep 11 19:53:43.653162 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:43.653064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:43.752442 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:43.752407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:43.859999 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:43.859965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:44.398873 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:44.398838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:44.416194 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:44.416150 2579 generic.go:356] "Generic (PLEG): container finished" podID="946c6c11-7caf-48b5-af4b-8471da3a498d" containerID="21e8214bdbccedff34df269b1a75bb5fe07558264480908152353e505d8ed448" exitCode=0 Sep 11 19:53:44.416368 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:44.416207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" event={"ID":"946c6c11-7caf-48b5-af4b-8471da3a498d","Type":"ContainerDied","Data":"21e8214bdbccedff34df269b1a75bb5fe07558264480908152353e505d8ed448"} Sep 11 19:53:45.070282 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:45.070253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:45.581575 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:45.581549 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" Sep 11 19:53:45.634249 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:45.634215 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c6c11-7caf-48b5-af4b-8471da3a498d-maas-api-service-ca\") pod \"946c6c11-7caf-48b5-af4b-8471da3a498d\" (UID: \"946c6c11-7caf-48b5-af4b-8471da3a498d\") " Sep 11 19:53:45.634583 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:45.634563 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946c6c11-7caf-48b5-af4b-8471da3a498d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "946c6c11-7caf-48b5-af4b-8471da3a498d" (UID: "946c6c11-7caf-48b5-af4b-8471da3a498d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:53:45.735073 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:45.734984 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c6c11-7caf-48b5-af4b-8471da3a498d-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:53:45.878869 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:45.878819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:46.444153 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:46.444112 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" Sep 11 19:53:46.444328 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:46.444159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx" event={"ID":"946c6c11-7caf-48b5-af4b-8471da3a498d","Type":"ContainerDied","Data":"c059dcd2554db0387f582cf7a68b539c6a1dcbf784af938201a4c507f7c3bdf4"} Sep 11 19:53:46.444328 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:46.444206 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c059dcd2554db0387f582cf7a68b539c6a1dcbf784af938201a4c507f7c3bdf4" Sep 11 19:53:46.605064 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:46.605029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:47.230110 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:47.230069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:47.924937 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:47.924904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:48.547115 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:48.547078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:48.649058 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:48.649028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:48.985465 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:48.985378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:49.389871 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:49.389837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:51.754641 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:51.754609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:52.342292 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:52.342221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:52.468098 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:52.468064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:52.886184 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:52.886152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:53.951216 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:53.951181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:54.893582 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:54.893550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:55.582875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.576893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:55.588710 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.588654 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx"] Sep 11 19:53:55.588960 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.588924 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" podUID="8f8e9ad1-be41-4c23-92e0-adec195d8421" containerName="payload-processing" containerID="cri-o://1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98" gracePeriod=30 Sep 11 19:53:55.665406 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.665374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:55.766765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.766695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:55.852182 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.852083 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx"] Sep 11 19:53:55.858333 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.858306 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a58a49-s22nx"] Sep 11 19:53:55.884645 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.884607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:55.925614 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.925591 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:53:55.975867 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:55.975414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:56.044905 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.044875 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7swj9\" (UniqueName: \"kubernetes.io/projected/8f8e9ad1-be41-4c23-92e0-adec195d8421-kube-api-access-7swj9\") pod \"8f8e9ad1-be41-4c23-92e0-adec195d8421\" (UID: \"8f8e9ad1-be41-4c23-92e0-adec195d8421\") " Sep 11 19:53:56.045073 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.044980 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e9ad1-be41-4c23-92e0-adec195d8421-plugins-config-volume\") pod \"8f8e9ad1-be41-4c23-92e0-adec195d8421\" (UID: \"8f8e9ad1-be41-4c23-92e0-adec195d8421\") " Sep 11 19:53:56.045359 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.045336 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8e9ad1-be41-4c23-92e0-adec195d8421-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f8e9ad1-be41-4c23-92e0-adec195d8421" (UID: "8f8e9ad1-be41-4c23-92e0-adec195d8421"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:53:56.047242 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.047218 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8e9ad1-be41-4c23-92e0-adec195d8421-kube-api-access-7swj9" (OuterVolumeSpecName: "kube-api-access-7swj9") pod "8f8e9ad1-be41-4c23-92e0-adec195d8421" (UID: "8f8e9ad1-be41-4c23-92e0-adec195d8421"). InnerVolumeSpecName "kube-api-access-7swj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:53:56.146110 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.146018 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e9ad1-be41-4c23-92e0-adec195d8421-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:53:56.146110 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.146052 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7swj9\" (UniqueName: \"kubernetes.io/projected/8f8e9ad1-be41-4c23-92e0-adec195d8421-kube-api-access-7swj9\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:53:56.198504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.198472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:56.507551 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.507465 2579 generic.go:356] "Generic (PLEG): container finished" podID="8f8e9ad1-be41-4c23-92e0-adec195d8421" containerID="1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98" exitCode=0 Sep 11 19:53:56.507551 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.507509 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" Sep 11 19:53:56.507551 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.507518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" event={"ID":"8f8e9ad1-be41-4c23-92e0-adec195d8421","Type":"ContainerDied","Data":"1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98"} Sep 11 19:53:56.507775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.507561 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx" event={"ID":"8f8e9ad1-be41-4c23-92e0-adec195d8421","Type":"ContainerDied","Data":"ae87bdc92a6622afb92fa1b834921a7e4038568d8435374e61e649d970395c1d"} Sep 11 19:53:56.507775 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.507582 2579 scope.go:117] "RemoveContainer" containerID="1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98" Sep 11 19:53:56.517877 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.517858 2579 scope.go:117] "RemoveContainer" containerID="1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98" Sep 11 19:53:56.518151 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:53:56.518135 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98\": container with ID starting with 1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98 not found: ID does not exist" containerID="1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98" Sep 11 19:53:56.518220 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.518160 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98"} err="failed to get container status \"1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98\": rpc error: code = NotFound desc = could not find container \"1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98\": container with ID starting with 1d2d8868a2f659f706c208512165c857c34ee269fbcee57fb457c9f0bbf4ce98 not found: ID does not exist" Sep 11 19:53:56.530594 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.530567 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx"] Sep 11 19:53:56.537270 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.537246 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a58a49-6875899b9-vs6fx"] Sep 11 19:53:56.782895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:56.782773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:57.394272 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:57.394237 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8e9ad1-be41-4c23-92e0-adec195d8421" path="/var/lib/kubelet/pods/8f8e9ad1-be41-4c23-92e0-adec195d8421/volumes" Sep 11 19:53:57.394585 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:57.394572 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="946c6c11-7caf-48b5-af4b-8471da3a498d" path="/var/lib/kubelet/pods/946c6c11-7caf-48b5-af4b-8471da3a498d/volumes" Sep 11 19:53:57.816031 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:57.815997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:58.425880 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.425838 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x"] Sep 11 19:53:58.426563 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.426543 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8e9ad1-be41-4c23-92e0-adec195d8421" containerName="payload-processing" Sep 11 19:53:58.426563 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.426562 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8e9ad1-be41-4c23-92e0-adec195d8421" containerName="payload-processing" Sep 11 19:53:58.426726 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.426597 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946c6c11-7caf-48b5-af4b-8471da3a498d" containerName="revoke-keys" Sep 11 19:53:58.426726 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.426606 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="946c6c11-7caf-48b5-af4b-8471da3a498d" containerName="revoke-keys" Sep 11 19:53:58.426726 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.426700 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="946c6c11-7caf-48b5-af4b-8471da3a498d" containerName="revoke-keys" Sep 11 19:53:58.426726 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.426713 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8e9ad1-be41-4c23-92e0-adec195d8421" containerName="payload-processing" Sep 11 19:53:58.431888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.431869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:58.436038 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.436019 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-105b25aa\"" Sep 11 19:53:58.436162 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.436103 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-105b25aa-dockercfg-k4cdh\"" Sep 11 19:53:58.447187 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.447155 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x"] Sep 11 19:53:58.490727 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.490445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49907290-7ea8-4ebc-a275-c262d3319106-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x\" (UID: \"49907290-7ea8-4ebc-a275-c262d3319106\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:58.490727 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.490554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bsp\" (UniqueName: \"kubernetes.io/projected/49907290-7ea8-4ebc-a275-c262d3319106-kube-api-access-d8bsp\") pod \"payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x\" (UID: \"49907290-7ea8-4ebc-a275-c262d3319106\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:58.592181 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.592138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49907290-7ea8-4ebc-a275-c262d3319106-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x\" (UID: \"49907290-7ea8-4ebc-a275-c262d3319106\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:58.592368 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.592225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bsp\" (UniqueName: \"kubernetes.io/projected/49907290-7ea8-4ebc-a275-c262d3319106-kube-api-access-d8bsp\") pod \"payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x\" (UID: \"49907290-7ea8-4ebc-a275-c262d3319106\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:58.592870 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.592850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49907290-7ea8-4ebc-a275-c262d3319106-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x\" (UID: \"49907290-7ea8-4ebc-a275-c262d3319106\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:58.602923 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.602895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bsp\" (UniqueName: \"kubernetes.io/projected/49907290-7ea8-4ebc-a275-c262d3319106-kube-api-access-d8bsp\") pod \"payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x\" (UID: \"49907290-7ea8-4ebc-a275-c262d3319106\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:58.744171 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.744071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:58.878624 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:58.878546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:59.050265 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:59.050238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x"] Sep 11 19:53:59.053206 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:53:59.053172 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49907290_7ea8_4ebc_a275_c262d3319106.slice/crio-e3e3ff79fb102851272f7195884420396d492fa713b1e7e4a3cda517b7899619 WatchSource:0}: Error finding container e3e3ff79fb102851272f7195884420396d492fa713b1e7e4a3cda517b7899619: Status 404 returned error can't find the container with id e3e3ff79fb102851272f7195884420396d492fa713b1e7e4a3cda517b7899619 Sep 11 19:53:59.418085 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:59.418039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:53:59.554263 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:59.554227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" event={"ID":"49907290-7ea8-4ebc-a275-c262d3319106","Type":"ContainerStarted","Data":"081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e"} Sep 11 19:53:59.554434 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:59.554288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" event={"ID":"49907290-7ea8-4ebc-a275-c262d3319106","Type":"ContainerStarted","Data":"e3e3ff79fb102851272f7195884420396d492fa713b1e7e4a3cda517b7899619"} Sep 11 19:53:59.554434 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:59.554316 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:53:59.575237 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:53:59.575177 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" podStartSLOduration=1.5751567309999999 podStartE2EDuration="1.575156731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:59.572385739 +0000 UTC m=+1472.705698342" watchObservedRunningTime="2026-09-11 19:53:59.575156731 +0000 UTC m=+1472.708469321" Sep 11 19:54:00.290817 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:00.290785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:00.984591 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:00.984561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:01.892923 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:01.892888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:02.447164 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.447089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:02.547939 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.547908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:02.574352 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.574322 2579 generic.go:356] "Generic (PLEG): container finished" podID="c206e47f-3c55-4953-ad34-34f1e242579e" containerID="ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3" exitCode=0 Sep 11 19:54:02.574544 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.574398 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" event={"ID":"c206e47f-3c55-4953-ad34-34f1e242579e","Type":"ContainerDied","Data":"ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3"} Sep 11 19:54:02.574872 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.574856 2579 scope.go:117] "RemoveContainer" containerID="ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3" Sep 11 19:54:02.576193 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.576162 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:02.576261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.576247 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:02.576298 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.576288 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:02.651552 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.651518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:02.713458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.713428 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t"] Sep 11 19:54:02.713664 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.713643 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" containerID="cri-o://e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1" gracePeriod=30 Sep 11 19:54:02.756784 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:02.756734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:03.261630 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.261588 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts"] Sep 11 19:54:03.265274 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.265240 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" Sep 11 19:54:03.277513 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.277475 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts"] Sep 11 19:54:03.278080 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.278041 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 19:54:03.279077 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.278380 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:54:03.352776 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.352734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bacaf9-92d0-45a2-807c-b1c7ac3133f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-04331c-9kkts\" (UID: \"84bacaf9-92d0-45a2-807c-b1c7ac3133f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" Sep 11 19:54:03.454165 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.454121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bacaf9-92d0-45a2-807c-b1c7ac3133f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-04331c-9kkts\" (UID: \"84bacaf9-92d0-45a2-807c-b1c7ac3133f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" Sep 11 19:54:03.454788 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.454761 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bacaf9-92d0-45a2-807c-b1c7ac3133f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-04331c-9kkts\" (UID: \"84bacaf9-92d0-45a2-807c-b1c7ac3133f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" Sep 11 19:54:03.584689 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.584640 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" Sep 11 19:54:03.588020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.587982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" event={"ID":"c206e47f-3c55-4953-ad34-34f1e242579e","Type":"ContainerStarted","Data":"3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a"} Sep 11 19:54:03.695795 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.695762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:03.872947 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.872912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts"] Sep 11 19:54:03.876055 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:54:03.876015 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bacaf9_92d0_45a2_807c_b1c7ac3133f3.slice/crio-ad454e6de988e9e9d5c0a9feb9c84b84919ad8a9799fb40cd4362689cd038711 WatchSource:0}: Error finding container ad454e6de988e9e9d5c0a9feb9c84b84919ad8a9799fb40cd4362689cd038711: Status 404 returned error can't find the container with id ad454e6de988e9e9d5c0a9feb9c84b84919ad8a9799fb40cd4362689cd038711 Sep 11 19:54:03.918927 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.918892 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:54:03.920389 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.920365 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 19:54:03.920492 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:03.920411 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 19:54:04.595497 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:04.595457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" event={"ID":"84bacaf9-92d0-45a2-807c-b1c7ac3133f3","Type":"ContainerStarted","Data":"425b9f2cc5b5a2347569eb1e04c3368c54268d1b14987b5929be81bdb839fde7"} Sep 11 19:54:04.595996 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:04.595510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" event={"ID":"84bacaf9-92d0-45a2-807c-b1c7ac3133f3","Type":"ContainerStarted","Data":"ad454e6de988e9e9d5c0a9feb9c84b84919ad8a9799fb40cd4362689cd038711"} Sep 11 19:54:04.616158 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:04.616099 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" podStartSLOduration=1.616081566 podStartE2EDuration="1.616081566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:04.615701163 +0000 UTC m=+1477.749013753" watchObservedRunningTime="2026-09-11 19:54:04.616081566 +0000 UTC m=+1477.749394155" Sep 11 19:54:04.668869 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:04.668810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:04.918605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:04.918574 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 19:54:04.918754 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:04.918631 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 19:54:05.603557 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:05.603517 2579 generic.go:356] "Generic (PLEG): container finished" podID="84bacaf9-92d0-45a2-807c-b1c7ac3133f3" containerID="425b9f2cc5b5a2347569eb1e04c3368c54268d1b14987b5929be81bdb839fde7" exitCode=0 Sep 11 19:54:05.603956 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:05.603601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" event={"ID":"84bacaf9-92d0-45a2-807c-b1c7ac3133f3","Type":"ContainerDied","Data":"425b9f2cc5b5a2347569eb1e04c3368c54268d1b14987b5929be81bdb839fde7"} Sep 11 19:54:05.616430 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:05.616399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:05.919315 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:05.919221 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 19:54:05.919315 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:05.919283 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 19:54:06.381422 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:06.381359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:06.809602 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:06.809578 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" Sep 11 19:54:06.919291 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:06.919257 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 19:54:06.919794 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:06.919750 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 19:54:06.999243 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:06.999149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bacaf9-92d0-45a2-807c-b1c7ac3133f3-maas-api-service-ca\") pod \"84bacaf9-92d0-45a2-807c-b1c7ac3133f3\" (UID: \"84bacaf9-92d0-45a2-807c-b1c7ac3133f3\") " Sep 11 19:54:06.999490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:06.999465 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bacaf9-92d0-45a2-807c-b1c7ac3133f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84bacaf9-92d0-45a2-807c-b1c7ac3133f3" (UID: "84bacaf9-92d0-45a2-807c-b1c7ac3133f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:54:07.100503 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.100463 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84bacaf9-92d0-45a2-807c-b1c7ac3133f3-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:07.251246 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.251164 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:54:07.278712 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.278671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:07.616315 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.616281 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" event={"ID":"84bacaf9-92d0-45a2-807c-b1c7ac3133f3","Type":"ContainerDied","Data":"ad454e6de988e9e9d5c0a9feb9c84b84919ad8a9799fb40cd4362689cd038711"} Sep 11 19:54:07.616315 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.616315 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts" Sep 11 19:54:07.616544 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.616337 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad454e6de988e9e9d5c0a9feb9c84b84919ad8a9799fb40cd4362689cd038711" Sep 11 19:54:07.919198 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.919163 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 19:54:07.919677 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.919224 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 19:54:07.936339 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:07.936306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:08.045017 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.044992 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:54:08.215012 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.214926 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-credential-socket\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215012 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.214980 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29576270-8f07-4992-b746-550d5d05b2f5-istio-podinfo\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215012 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215013 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-data\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215257 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215053 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-socket\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215257 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215090 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpmgw\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-kube-api-access-tpmgw\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215257 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215138 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-istio-token\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215257 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215189 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-certs\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215257 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-envoy\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215257 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215226 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:54:08.215532 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215265 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:54:08.215532 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215280 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29576270-8f07-4992-b746-550d5d05b2f5-istiod-ca-cert\") pod \"29576270-8f07-4992-b746-550d5d05b2f5\" (UID: \"29576270-8f07-4992-b746-550d5d05b2f5\") " Sep 11 19:54:08.215532 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215316 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:54:08.215691 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215621 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:54:08.215765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215735 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.215765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215758 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-credential-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.215903 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215804 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-data\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.215903 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215852 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-workload-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.216038 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.215938 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29576270-8f07-4992-b746-550d5d05b2f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:54:08.217561 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.217536 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:54:08.217672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.217560 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:54:08.217853 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.217818 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29576270-8f07-4992-b746-550d5d05b2f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 19:54:08.217913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.217879 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-kube-api-access-tpmgw" (OuterVolumeSpecName: "kube-api-access-tpmgw") pod "29576270-8f07-4992-b746-550d5d05b2f5" (UID: "29576270-8f07-4992-b746-550d5d05b2f5"). InnerVolumeSpecName "kube-api-access-tpmgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:54:08.317266 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.317228 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29576270-8f07-4992-b746-550d5d05b2f5-istiod-ca-cert\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.317266 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.317262 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29576270-8f07-4992-b746-550d5d05b2f5-istio-podinfo\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.317266 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.317272 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmgw\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-kube-api-access-tpmgw\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.317514 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.317283 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29576270-8f07-4992-b746-550d5d05b2f5-istio-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.317514 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.317292 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29576270-8f07-4992-b746-550d5d05b2f5-istio-envoy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:08.624521 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.624489 2579 generic.go:356] "Generic (PLEG): container finished" podID="29576270-8f07-4992-b746-550d5d05b2f5" containerID="e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1" exitCode=0 Sep 11 19:54:08.624702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.624540 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" Sep 11 19:54:08.624702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.624571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" event={"ID":"29576270-8f07-4992-b746-550d5d05b2f5","Type":"ContainerDied","Data":"e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1"} Sep 11 19:54:08.624702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.624624 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t" event={"ID":"29576270-8f07-4992-b746-550d5d05b2f5","Type":"ContainerDied","Data":"f740739962192a831569d18ba729968db36b3356b0806e74dc26817311e88c31"} Sep 11 19:54:08.624702 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.624645 2579 scope.go:117] "RemoveContainer" containerID="e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1" Sep 11 19:54:08.634837 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.634800 2579 scope.go:117] "RemoveContainer" containerID="8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4" Sep 11 19:54:08.645299 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.645277 2579 scope.go:117] "RemoveContainer" containerID="e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1" Sep 11 19:54:08.645651 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:54:08.645627 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1\": container with ID starting with e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1 not found: ID does not exist" containerID="e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1" Sep 11 19:54:08.645704 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.645665 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1"} err="failed to get container status \"e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1\": rpc error: code = NotFound desc = could not find container \"e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1\": container with ID starting with e86429d088456d2275c4dd9b57283f7d64cd6ecdc907860219cb4a33e3e096d1 not found: ID does not exist" Sep 11 19:54:08.645704 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.645691 2579 scope.go:117] "RemoveContainer" containerID="8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4" Sep 11 19:54:08.645988 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:54:08.645959 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4\": container with ID starting with 8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4 not found: ID does not exist" containerID="8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4" Sep 11 19:54:08.646028 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.645997 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4"} err="failed to get container status \"8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4\": rpc error: code = NotFound desc = could not find container \"8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4\": container with ID starting with 8b8697577a03c21e768664b409dc7fc7f25a1abea140f4630f0b3ac99687cee4 not found: ID does not exist" Sep 11 19:54:08.647888 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.647866 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t"] Sep 11 19:54:08.650130 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.650112 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a58a49-openshift-default-6d88887494-xn58t"] Sep 11 19:54:08.719432 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.719399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:08.922923 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.922815 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:54:08.923894 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:08.923873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 19:54:09.394723 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:09.394685 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29576270-8f07-4992-b746-550d5d05b2f5" path="/var/lib/kubelet/pods/29576270-8f07-4992-b746-550d5d05b2f5/volumes" Sep 11 19:54:09.487801 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:09.487774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:10.198322 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:10.198292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:10.563810 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:10.563779 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:54:11.194180 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:11.194134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:12.014402 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:12.014370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:12.118637 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:12.118596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:12.794123 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:12.794091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:13.808944 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:13.808904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:14.648716 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:14.648647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:15.564668 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.564621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:15.607157 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.607126 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64"] Sep 11 19:54:15.607371 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.607344 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" podUID="ac433d43-277e-4665-9b32-a0fc9fc3a2cb" containerName="payload-processing" containerID="cri-o://6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585" gracePeriod=30 Sep 11 19:54:15.621306 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.621280 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2"] Sep 11 19:54:15.621524 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.621498 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" podUID="2befbe77-3c01-4336-aaef-1de2199aa0ad" containerName="payload-pre-processing" containerID="cri-o://010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d" gracePeriod=30 Sep 11 19:54:15.662689 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.662655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:15.772331 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.772301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:15.829107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.828982 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts"] Sep 11 19:54:15.837944 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.837895 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-04331c-9kkts"] Sep 11 19:54:15.863174 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.863105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:15.928429 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.928402 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:54:15.931429 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.931410 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:54:15.997352 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.997321 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-plugins-config-volume\") pod \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\" (UID: \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\") " Sep 11 19:54:15.997528 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.997394 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2befbe77-3c01-4336-aaef-1de2199aa0ad-plugins-config-volume\") pod \"2befbe77-3c01-4336-aaef-1de2199aa0ad\" (UID: \"2befbe77-3c01-4336-aaef-1de2199aa0ad\") " Sep 11 19:54:15.997528 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.997431 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqwpn\" (UniqueName: \"kubernetes.io/projected/2befbe77-3c01-4336-aaef-1de2199aa0ad-kube-api-access-qqwpn\") pod \"2befbe77-3c01-4336-aaef-1de2199aa0ad\" (UID: \"2befbe77-3c01-4336-aaef-1de2199aa0ad\") " Sep 11 19:54:15.997612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.997531 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psq75\" (UniqueName: \"kubernetes.io/projected/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-kube-api-access-psq75\") pod \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\" (UID: \"ac433d43-277e-4665-9b32-a0fc9fc3a2cb\") " Sep 11 19:54:15.997754 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.997727 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac433d43-277e-4665-9b32-a0fc9fc3a2cb" (UID: "ac433d43-277e-4665-9b32-a0fc9fc3a2cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:54:15.997819 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.997793 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2befbe77-3c01-4336-aaef-1de2199aa0ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2befbe77-3c01-4336-aaef-1de2199aa0ad" (UID: "2befbe77-3c01-4336-aaef-1de2199aa0ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:54:15.997961 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.997946 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:15.998020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.997963 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2befbe77-3c01-4336-aaef-1de2199aa0ad-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:15.999937 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.999913 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-kube-api-access-psq75" (OuterVolumeSpecName: "kube-api-access-psq75") pod "ac433d43-277e-4665-9b32-a0fc9fc3a2cb" (UID: "ac433d43-277e-4665-9b32-a0fc9fc3a2cb"). InnerVolumeSpecName "kube-api-access-psq75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:54:16.000011 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:15.999985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2befbe77-3c01-4336-aaef-1de2199aa0ad-kube-api-access-qqwpn" (OuterVolumeSpecName: "kube-api-access-qqwpn") pod "2befbe77-3c01-4336-aaef-1de2199aa0ad" (UID: "2befbe77-3c01-4336-aaef-1de2199aa0ad"). InnerVolumeSpecName "kube-api-access-qqwpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:54:16.099380 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.099294 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psq75\" (UniqueName: \"kubernetes.io/projected/ac433d43-277e-4665-9b32-a0fc9fc3a2cb-kube-api-access-psq75\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:16.099380 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.099324 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwpn\" (UniqueName: \"kubernetes.io/projected/2befbe77-3c01-4336-aaef-1de2199aa0ad-kube-api-access-qqwpn\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:54:16.169750 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.169718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:16.283233 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.283195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:16.677294 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.677253 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac433d43-277e-4665-9b32-a0fc9fc3a2cb" containerID="6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585" exitCode=0 Sep 11 19:54:16.677758 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.677298 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" Sep 11 19:54:16.677758 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.677339 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" event={"ID":"ac433d43-277e-4665-9b32-a0fc9fc3a2cb","Type":"ContainerDied","Data":"6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585"} Sep 11 19:54:16.677758 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.677402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64" event={"ID":"ac433d43-277e-4665-9b32-a0fc9fc3a2cb","Type":"ContainerDied","Data":"e35c24a226042a170352f2b6f7982332fc8899ea7311633c2dfc0c4cd0d68dee"} Sep 11 19:54:16.677758 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.677434 2579 scope.go:117] "RemoveContainer" containerID="6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585" Sep 11 19:54:16.678813 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.678788 2579 generic.go:356] "Generic (PLEG): container finished" podID="2befbe77-3c01-4336-aaef-1de2199aa0ad" containerID="010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d" exitCode=0 Sep 11 19:54:16.678912 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.678853 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" Sep 11 19:54:16.678912 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.678890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" event={"ID":"2befbe77-3c01-4336-aaef-1de2199aa0ad","Type":"ContainerDied","Data":"010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d"} Sep 11 19:54:16.679030 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.678927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2" event={"ID":"2befbe77-3c01-4336-aaef-1de2199aa0ad","Type":"ContainerDied","Data":"046303435ae79d2a63561d594e15c2a81cec19053a7777c58306278080536a9d"} Sep 11 19:54:16.689532 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.689507 2579 scope.go:117] "RemoveContainer" containerID="6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585" Sep 11 19:54:16.690009 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:54:16.689976 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585\": container with ID starting with 6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585 not found: ID does not exist" containerID="6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585" Sep 11 19:54:16.690104 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.690020 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585"} err="failed to get container status \"6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585\": rpc error: code = NotFound desc = could not find container \"6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585\": container with ID starting with 6f28653774e919dad5a5152965bdf4c3382ea6c5c6362a28043a491919b33585 not found: ID does not exist" Sep 11 19:54:16.690104 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.690043 2579 scope.go:117] "RemoveContainer" containerID="010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d" Sep 11 19:54:16.699777 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.699753 2579 scope.go:117] "RemoveContainer" containerID="010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d" Sep 11 19:54:16.700194 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:54:16.700158 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d\": container with ID starting with 010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d not found: ID does not exist" containerID="010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d" Sep 11 19:54:16.700270 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.700216 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d"} err="failed to get container status \"010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d\": rpc error: code = NotFound desc = could not find container \"010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d\": container with ID starting with 010e050739f23616fbab321472bf6cc359de9777954a80165f6deaf1090c039d not found: ID does not exist" Sep 11 19:54:16.712016 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.711994 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64"] Sep 11 19:54:16.718757 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.718713 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04331c-f7d479cf-42r64"] Sep 11 19:54:16.740759 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.740728 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2"] Sep 11 19:54:16.749556 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:16.749526 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04331c-577fd75d54-hwss2"] Sep 11 19:54:17.112748 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:17.112721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:17.372796 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:17.372711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:17.396319 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:17.396281 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2befbe77-3c01-4336-aaef-1de2199aa0ad" path="/var/lib/kubelet/pods/2befbe77-3c01-4336-aaef-1de2199aa0ad/volumes" Sep 11 19:54:17.396737 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:17.396717 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84bacaf9-92d0-45a2-807c-b1c7ac3133f3" path="/var/lib/kubelet/pods/84bacaf9-92d0-45a2-807c-b1c7ac3133f3/volumes" Sep 11 19:54:17.397143 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:17.397125 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac433d43-277e-4665-9b32-a0fc9fc3a2cb" path="/var/lib/kubelet/pods/ac433d43-277e-4665-9b32-a0fc9fc3a2cb/volumes" Sep 11 19:54:18.406254 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:18.406210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:19.284529 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:19.284496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:20.011657 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:20.011619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:20.973311 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:20.973260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:22.484287 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:22.484259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:22.584436 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:22.584402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:22.742611 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:22.742519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:23.291595 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.291555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:23.569401 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.569304 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95"] Sep 11 19:54:23.569990 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.569970 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" Sep 11 19:54:23.569990 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.569992 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" Sep 11 19:54:23.570201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570028 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2befbe77-3c01-4336-aaef-1de2199aa0ad" containerName="payload-pre-processing" Sep 11 19:54:23.570201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570037 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2befbe77-3c01-4336-aaef-1de2199aa0ad" containerName="payload-pre-processing" Sep 11 19:54:23.570201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570050 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84bacaf9-92d0-45a2-807c-b1c7ac3133f3" containerName="revoke-keys" Sep 11 19:54:23.570201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570059 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="84bacaf9-92d0-45a2-807c-b1c7ac3133f3" containerName="revoke-keys" Sep 11 19:54:23.570201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570073 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac433d43-277e-4665-9b32-a0fc9fc3a2cb" containerName="payload-processing" Sep 11 19:54:23.570201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570081 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac433d43-277e-4665-9b32-a0fc9fc3a2cb" containerName="payload-processing" Sep 11 19:54:23.570605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570212 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" Sep 11 19:54:23.570605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570225 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" Sep 11 19:54:23.570605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570263 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2befbe77-3c01-4336-aaef-1de2199aa0ad" containerName="payload-pre-processing" Sep 11 19:54:23.570605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570278 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac433d43-277e-4665-9b32-a0fc9fc3a2cb" containerName="payload-processing" Sep 11 19:54:23.570605 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.570294 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="84bacaf9-92d0-45a2-807c-b1c7ac3133f3" containerName="revoke-keys" Sep 11 19:54:23.573664 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.573641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.576449 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.576428 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-57d97d-openshift-default-dockercfg-j4n4j\"" Sep 11 19:54:23.589201 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.589177 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95"] Sep 11 19:54:23.605117 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.605088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-certs\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-envoy\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-podinfo\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l598c\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-kube-api-access-l598c\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istiod-ca-cert\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-socket\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-credential-socket\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-token\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.681669 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.681566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-data\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.762231 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.762201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:23.783015 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.782984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l598c\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-kube-api-access-l598c\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istiod-ca-cert\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-socket\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-credential-socket\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-token\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783192 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-data\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-certs\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-envoy\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783496 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-podinfo\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783611 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-socket\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783611 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-credential-socket\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783768 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istiod-ca-cert\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783895 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-data\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.783966 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.783896 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-certs\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.785921 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.785894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-envoy\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.786014 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.785963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-podinfo\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.792137 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.792104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-token\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.792222 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.792192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l598c\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-kube-api-access-l598c\") pod \"e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:23.884760 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:23.884670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:24.166230 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:24.166206 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95"] Sep 11 19:54:24.168051 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:54:24.168022 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45eca849_ed2b_493f_8c8f_5d04ec610ee9.slice/crio-f86ea23e22122c12f997f0cc185ed0c5114f5f386293db6293da42ce78d88290 WatchSource:0}: Error finding container f86ea23e22122c12f997f0cc185ed0c5114f5f386293db6293da42ce78d88290: Status 404 returned error can't find the container with id f86ea23e22122c12f997f0cc185ed0c5114f5f386293db6293da42ce78d88290 Sep 11 19:54:24.170599 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:24.170565 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:24.170692 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:24.170672 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:24.170765 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:24.170724 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:24.735580 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:24.735544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" event={"ID":"45eca849-ed2b-493f-8c8f-5d04ec610ee9","Type":"ContainerStarted","Data":"a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925"} Sep 11 19:54:24.736012 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:24.735603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" event={"ID":"45eca849-ed2b-493f-8c8f-5d04ec610ee9","Type":"ContainerStarted","Data":"f86ea23e22122c12f997f0cc185ed0c5114f5f386293db6293da42ce78d88290"} Sep 11 19:54:24.767321 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:24.767272 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" podStartSLOduration=1.767256133 podStartE2EDuration="1.767256133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:24.76544542 +0000 UTC m=+1497.898758009" watchObservedRunningTime="2026-09-11 19:54:24.767256133 +0000 UTC m=+1497.900568722" Sep 11 19:54:24.884917 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:24.884876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:25.891059 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:25.891031 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:25.984756 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:25.984725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:26.670491 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:26.670441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:26.747455 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:26.747418 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:26.748491 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:26.748469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:26.771740 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:26.771705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:27.441027 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:27.441005 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:54:27.442458 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:27.442438 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:54:27.459952 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:27.459928 2579 scope.go:117] "RemoveContainer" containerID="3608eeb35917d14a86ef4b56fabe46cf7981194e6f49d9e92a9c06ea02bac61f" Sep 11 19:54:27.469419 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:27.469398 2579 scope.go:117] "RemoveContainer" containerID="cbbdd5b989bf0a1e8a8e849fd734e95c72bdd75041257487151db7e373e79cee" Sep 11 19:54:27.788902 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:27.788861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:28.691806 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:28.691778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:29.572514 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:29.572484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:30.249642 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:30.249608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:30.953980 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:30.953948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:31.056149 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.056085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:31.518639 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.518611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:31.609849 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.609779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:31.611282 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.611258 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5"] Sep 11 19:54:31.611798 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.611780 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" Sep 11 19:54:31.611918 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.611800 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29576270-8f07-4992-b746-550d5d05b2f5" containerName="istio-proxy" Sep 11 19:54:31.617035 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.617018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:31.619793 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.619776 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-57d97d\"" Sep 11 19:54:31.626891 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.626869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5"] Sep 11 19:54:31.647078 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.647044 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf"] Sep 11 19:54:31.651030 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.651012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:31.653886 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.653865 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-57d97d-dockercfg-ts6hn\"" Sep 11 19:54:31.661451 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.661427 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf"] Sep 11 19:54:31.771050 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.770898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkxd\" (UniqueName: \"kubernetes.io/projected/919b4976-acd0-4389-bf35-e1ffdc88b287-kube-api-access-xlkxd\") pod \"payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf\" (UID: \"919b4976-acd0-4389-bf35-e1ffdc88b287\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:31.771233 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.771068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5\" (UID: \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:31.771233 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.771166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsnj\" (UniqueName: \"kubernetes.io/projected/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-kube-api-access-dgsnj\") pod \"payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5\" (UID: \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:31.771346 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.771234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919b4976-acd0-4389-bf35-e1ffdc88b287-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf\" (UID: \"919b4976-acd0-4389-bf35-e1ffdc88b287\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:31.872737 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.872703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919b4976-acd0-4389-bf35-e1ffdc88b287-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf\" (UID: \"919b4976-acd0-4389-bf35-e1ffdc88b287\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:31.872945 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.872761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkxd\" (UniqueName: \"kubernetes.io/projected/919b4976-acd0-4389-bf35-e1ffdc88b287-kube-api-access-xlkxd\") pod \"payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf\" (UID: \"919b4976-acd0-4389-bf35-e1ffdc88b287\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:31.872945 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.872804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5\" (UID: \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:31.872945 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.872886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsnj\" (UniqueName: \"kubernetes.io/projected/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-kube-api-access-dgsnj\") pod \"payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5\" (UID: \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:31.873550 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.873530 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919b4976-acd0-4389-bf35-e1ffdc88b287-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf\" (UID: \"919b4976-acd0-4389-bf35-e1ffdc88b287\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:31.873550 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.873544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5\" (UID: \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:31.882123 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.882095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsnj\" (UniqueName: \"kubernetes.io/projected/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-kube-api-access-dgsnj\") pod \"payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5\" (UID: \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:31.882251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.882171 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkxd\" (UniqueName: \"kubernetes.io/projected/919b4976-acd0-4389-bf35-e1ffdc88b287-kube-api-access-xlkxd\") pod \"payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf\" (UID: \"919b4976-acd0-4389-bf35-e1ffdc88b287\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:31.929239 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.929202 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:31.962644 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:31.962611 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:32.204881 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.204854 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5"] Sep 11 19:54:32.206764 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:54:32.206739 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec57c7e_70a9_44f1_9f67_664f9fe2fd15.slice/crio-1156db301fdf23f65c5e17770e4966b7f7289ea65978179546d2b8ed29423fa3 WatchSource:0}: Error finding container 1156db301fdf23f65c5e17770e4966b7f7289ea65978179546d2b8ed29423fa3: Status 404 returned error can't find the container with id 1156db301fdf23f65c5e17770e4966b7f7289ea65978179546d2b8ed29423fa3 Sep 11 19:54:32.252792 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.252767 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf"] Sep 11 19:54:32.255441 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:54:32.255414 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919b4976_acd0_4389_bf35_e1ffdc88b287.slice/crio-dd35c464491f02bbedacff48a07aae3585ab9b6f8d70a2390dde50b2390c3d42 WatchSource:0}: Error finding container dd35c464491f02bbedacff48a07aae3585ab9b6f8d70a2390dde50b2390c3d42: Status 404 returned error can't find the container with id dd35c464491f02bbedacff48a07aae3585ab9b6f8d70a2390dde50b2390c3d42 Sep 11 19:54:32.794379 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.794340 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" event={"ID":"919b4976-acd0-4389-bf35-e1ffdc88b287","Type":"ContainerStarted","Data":"5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c"} Sep 11 19:54:32.794927 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.794406 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" event={"ID":"919b4976-acd0-4389-bf35-e1ffdc88b287","Type":"ContainerStarted","Data":"dd35c464491f02bbedacff48a07aae3585ab9b6f8d70a2390dde50b2390c3d42"} Sep 11 19:54:32.794927 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.794436 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:32.796216 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.796190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" event={"ID":"fec57c7e-70a9-44f1-9f67-664f9fe2fd15","Type":"ContainerStarted","Data":"42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863"} Sep 11 19:54:32.796342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.796244 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:32.796342 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.796268 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" event={"ID":"fec57c7e-70a9-44f1-9f67-664f9fe2fd15","Type":"ContainerStarted","Data":"1156db301fdf23f65c5e17770e4966b7f7289ea65978179546d2b8ed29423fa3"} Sep 11 19:54:32.814861 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.814782 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" podStartSLOduration=1.814760917 podStartE2EDuration="1.814760917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:32.812022128 +0000 UTC m=+1505.945334717" watchObservedRunningTime="2026-09-11 19:54:32.814760917 +0000 UTC m=+1505.948073509" Sep 11 19:54:32.830712 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:32.830658 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" podStartSLOduration=1.830642846 podStartE2EDuration="1.830642846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:32.828222121 +0000 UTC m=+1505.961534711" watchObservedRunningTime="2026-09-11 19:54:32.830642846 +0000 UTC m=+1505.963955434" Sep 11 19:54:43.806049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:43.806015 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 19:54:43.806474 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:43.806324 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 19:54:44.959590 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:44.959555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:50.941273 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:50.941236 2579 generic.go:356] "Generic (PLEG): container finished" podID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerID="a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925" exitCode=0 Sep 11 19:54:50.941703 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:50.941311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" event={"ID":"45eca849-ed2b-493f-8c8f-5d04ec610ee9","Type":"ContainerDied","Data":"a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925"} Sep 11 19:54:50.941882 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:50.941865 2579 scope.go:117] "RemoveContainer" containerID="a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925" Sep 11 19:54:50.943071 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:50.943039 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:50.943181 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:50.943125 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:50.943181 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:50.943173 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:51.948985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:51.948949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" event={"ID":"45eca849-ed2b-493f-8c8f-5d04ec610ee9","Type":"ContainerStarted","Data":"22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b"} Sep 11 19:54:52.885107 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:52.885039 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:52.886469 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:52.886443 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 11 19:54:52.886604 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:52.886496 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 11 19:54:53.083744 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:53.083702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:53.885199 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:53.885156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:53.885199 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:53.885172 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 11 19:54:53.885497 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:53.885222 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 11 19:54:54.615261 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:54.615228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:54.885899 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:54.885798 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 11 19:54:54.885899 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:54.885883 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 11 19:54:55.285725 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:55.285682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:55.885338 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:55.885295 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 11 19:54:55.885981 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:55.885928 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 11 19:54:56.164560 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:56.164441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:56.740615 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:56.740582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:56.747487 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:56.747462 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:56.851474 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:56.851446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:56.885804 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:56.885767 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 11 19:54:56.886264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:56.885874 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 11 19:54:57.074731 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.074693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:57.396711 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.396632 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w"] Sep 11 19:54:57.400943 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.400921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.409015 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.408993 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8a4eb7-openshift-default-dockercfg-t5bzv\"" Sep 11 19:54:57.432049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.432020 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w"] Sep 11 19:54:57.451897 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.451859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:57.552325 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j2zk\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-kube-api-access-9j2zk\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.552325 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-data\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.552572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-credential-socket\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.552572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f873fa-6643-4570-94e7-c121820936c5-istio-podinfo\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.552572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-certs\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.552572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f873fa-6643-4570-94e7-c121820936c5-istiod-ca-cert\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.552572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-istio-token\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.552572 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-envoy\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.552923 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.552590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-socket\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654197 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-socket\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654197 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j2zk\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-kube-api-access-9j2zk\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654197 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-data\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654511 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-credential-socket\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654511 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f873fa-6643-4570-94e7-c121820936c5-istio-podinfo\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654511 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-certs\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654511 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f873fa-6643-4570-94e7-c121820936c5-istiod-ca-cert\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654511 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-istio-token\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654511 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-envoy\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654890 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-socket\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654890 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654624 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-data\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654890 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-credential-socket\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.654890 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.654811 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-certs\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.655384 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.655361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f873fa-6643-4570-94e7-c121820936c5-istiod-ca-cert\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.657450 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.657426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-envoy\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.657592 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.657573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f873fa-6643-4570-94e7-c121820936c5-istio-podinfo\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.663854 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.663790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-istio-token\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.664012 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.663970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j2zk\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-kube-api-access-9j2zk\") pod \"e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.707724 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.707686 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:57.711750 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.711728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:57.889337 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.889310 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:57.890603 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:57.890586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 19:54:58.014420 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:58.014392 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w"] Sep 11 19:54:58.016184 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:54:58.016160 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f873fa_6643_4570_94e7_c121820936c5.slice/crio-8808f43f900c3ee78f70948429ee19d791bfc627baf6799179ab0dcf211dc162 WatchSource:0}: Error finding container 8808f43f900c3ee78f70948429ee19d791bfc627baf6799179ab0dcf211dc162: Status 404 returned error can't find the container with id 8808f43f900c3ee78f70948429ee19d791bfc627baf6799179ab0dcf211dc162 Sep 11 19:54:58.018614 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:58.018577 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:58.018706 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:58.018662 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:58.018748 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:58.018702 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:54:58.143294 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:58.143232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:54:59.020341 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:59.020301 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" event={"ID":"13f873fa-6643-4570-94e7-c121820936c5","Type":"ContainerStarted","Data":"b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13"} Sep 11 19:54:59.020766 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:59.020386 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" event={"ID":"13f873fa-6643-4570-94e7-c121820936c5","Type":"ContainerStarted","Data":"8808f43f900c3ee78f70948429ee19d791bfc627baf6799179ab0dcf211dc162"} Sep 11 19:54:59.043090 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:59.042927 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" podStartSLOduration=2.042908876 podStartE2EDuration="2.042908876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:59.041958879 +0000 UTC m=+1532.175271459" watchObservedRunningTime="2026-09-11 19:54:59.042908876 +0000 UTC m=+1532.176221464" Sep 11 19:54:59.712916 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:59.712881 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:54:59.718154 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:54:59.718118 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:55:00.025936 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:00.025854 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:55:00.026638 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:00.026622 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:55:00.294741 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:00.294710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:01.060902 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:01.060864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:01.207875 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:01.207849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:01.308876 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:01.308838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:02.105215 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:02.105189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:02.995292 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:02.995265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:03.675313 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:03.675285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:04.375961 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:04.375921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:04.676566 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:04.676475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:05.181153 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:05.181124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:05.817885 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:05.817855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:05.918408 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:05.918377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:05.982357 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:05.982324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:06.279563 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.279472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:06.374399 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.374361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:06.391790 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.391746 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb"] Sep 11 19:55:06.396713 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.396692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:06.400808 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.400790 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8a4eb7-dockercfg-nlc4s\"" Sep 11 19:55:06.401385 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.401357 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8a4eb7\"" Sep 11 19:55:06.410531 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.410508 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb"] Sep 11 19:55:06.557020 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.556984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk47g\" (UniqueName: \"kubernetes.io/projected/64a49f85-9a6c-4cc9-b629-4cbf83297d39-kube-api-access-bk47g\") pod \"payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb\" (UID: \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:06.557213 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.557052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a49f85-9a6c-4cc9-b629-4cbf83297d39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb\" (UID: \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:06.658418 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.658383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a49f85-9a6c-4cc9-b629-4cbf83297d39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb\" (UID: \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:06.658599 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.658525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk47g\" (UniqueName: \"kubernetes.io/projected/64a49f85-9a6c-4cc9-b629-4cbf83297d39-kube-api-access-bk47g\") pod \"payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb\" (UID: \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:06.659151 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.659133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a49f85-9a6c-4cc9-b629-4cbf83297d39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb\" (UID: \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:06.666729 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.666698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk47g\" (UniqueName: \"kubernetes.io/projected/64a49f85-9a6c-4cc9-b629-4cbf83297d39-kube-api-access-bk47g\") pod \"payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb\" (UID: \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:06.709506 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.709465 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:06.985998 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:06.985974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb"] Sep 11 19:55:06.988397 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:55:06.988366 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a49f85_9a6c_4cc9_b629_4cbf83297d39.slice/crio-7fc3663af86f152f48e829297d0a1fd2edb2e7019ec98e85c9ddcfb73d6e1ba5 WatchSource:0}: Error finding container 7fc3663af86f152f48e829297d0a1fd2edb2e7019ec98e85c9ddcfb73d6e1ba5: Status 404 returned error can't find the container with id 7fc3663af86f152f48e829297d0a1fd2edb2e7019ec98e85c9ddcfb73d6e1ba5 Sep 11 19:55:07.072045 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:07.071999 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" event={"ID":"64a49f85-9a6c-4cc9-b629-4cbf83297d39","Type":"ContainerStarted","Data":"86dead006f0390a934614ff6977936374175d749188ff7fa40a93ebc525efa97"} Sep 11 19:55:07.072252 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:07.072069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" event={"ID":"64a49f85-9a6c-4cc9-b629-4cbf83297d39","Type":"ContainerStarted","Data":"7fc3663af86f152f48e829297d0a1fd2edb2e7019ec98e85c9ddcfb73d6e1ba5"} Sep 11 19:55:07.072252 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:07.072100 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:07.092302 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:07.092248 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" podStartSLOduration=1.0922310020000001 podStartE2EDuration="1.092231002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:55:07.088818675 +0000 UTC m=+1540.222131277" watchObservedRunningTime="2026-09-11 19:55:07.092231002 +0000 UTC m=+1540.225543590" Sep 11 19:55:07.220595 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:07.220559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:08.142672 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:08.142643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:11.028077 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:11.028041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:18.081195 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:18.081159 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 19:55:27.520791 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:27.520750 2579 scope.go:117] "RemoveContainer" containerID="91b506f6cc3928a79de049adc2699adfad61651717c008d857307647d5917432" Sep 11 19:55:27.625393 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:27.625365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:28.392802 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:28.392773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:29.076544 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:29.076511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:29.869913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:29.869871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:30.519424 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:30.519378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:31.031464 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:31.031429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:31.186466 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:31.186436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:31.918152 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:31.918123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:32.770145 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:32.770113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:34.256094 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:34.255984 2579 generic.go:356] "Generic (PLEG): container finished" podID="13f873fa-6643-4570-94e7-c121820936c5" containerID="b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13" exitCode=0 Sep 11 19:55:34.256094 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:34.256060 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" event={"ID":"13f873fa-6643-4570-94e7-c121820936c5","Type":"ContainerDied","Data":"b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13"} Sep 11 19:55:34.256592 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:34.256525 2579 scope.go:117] "RemoveContainer" containerID="b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13" Sep 11 19:55:34.257869 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:34.257806 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:55:34.257950 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:34.257927 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:55:34.258000 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:34.257975 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 19:55:35.263891 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:35.263853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" event={"ID":"13f873fa-6643-4570-94e7-c121820936c5","Type":"ContainerStarted","Data":"ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249"} Sep 11 19:55:35.711944 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:35.711900 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:55:35.712247 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:35.712221 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 19:55:35.712344 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:35.712281 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 19:55:36.712781 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:36.712742 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 19:55:36.713208 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:36.712802 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 19:55:37.712628 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:37.712583 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 19:55:37.712884 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:37.712662 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 19:55:38.713090 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:38.713055 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 19:55:38.713504 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:38.713113 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 19:55:39.713134 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:39.713089 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 19:55:39.713611 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:39.713174 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 19:55:40.712251 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:40.712208 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 19:55:40.712442 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:40.712268 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 19:55:41.717200 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:41.717170 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:55:41.717599 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:41.717558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:55:41.718228 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:41.718209 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 19:55:42.176919 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:42.176889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:42.323340 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:42.323309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:42.432667 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:42.432585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:47.456275 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:47.456239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:48.991769 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:48.991737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:51.869771 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:51.869735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:55:51.926023 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:55:51.925990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:56:04.525049 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:56:04.525018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:56:19.751730 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:56:19.751647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:56:24.465441 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:56:24.465408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:56:27.543101 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:56:27.543068 2579 scope.go:117] "RemoveContainer" containerID="1a1bb1e418f22dd729f2b78ae5c37cbf84baf5bd59fe31877a49df7da0e2b702" Sep 11 19:56:27.552209 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:56:27.552182 2579 scope.go:117] "RemoveContainer" containerID="4ad44a9cac90804e7883204983e75f41b19cba060125c7c0e259761b6b58fafe" Sep 11 19:56:57.991939 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:56:57.991909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:56:58.052494 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:56:58.052464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:01.371264 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:01.371229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:06.417279 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:06.417249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:16.909758 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:16.909724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:22.063292 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:22.063260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:25.765963 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:25.765933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:26.567473 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:26.567441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:26.774318 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:26.774284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:27.468254 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:27.468224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:28.093683 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:28.093649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:28.879447 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:28.879395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:29.682161 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:29.682129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:30.405016 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:30.404986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:31.047532 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:31.047504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:31.785490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:31.785448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:32.289986 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:32.289956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:41.006738 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.006696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:41.043986 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.043954 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x"] Sep 11 19:57:41.044244 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.044215 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" podUID="49907290-7ea8-4ebc-a275-c262d3319106" containerName="payload-pre-processing" containerID="cri-o://081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e" gracePeriod=30 Sep 11 19:57:41.103594 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.103557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:41.207127 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.207088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:41.269985 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.269694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:41.351954 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.351929 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:57:41.473119 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.473083 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8bsp\" (UniqueName: \"kubernetes.io/projected/49907290-7ea8-4ebc-a275-c262d3319106-kube-api-access-d8bsp\") pod \"49907290-7ea8-4ebc-a275-c262d3319106\" (UID: \"49907290-7ea8-4ebc-a275-c262d3319106\") " Sep 11 19:57:41.473300 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.473149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49907290-7ea8-4ebc-a275-c262d3319106-plugins-config-volume\") pod \"49907290-7ea8-4ebc-a275-c262d3319106\" (UID: \"49907290-7ea8-4ebc-a275-c262d3319106\") " Sep 11 19:57:41.473547 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.473524 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49907290-7ea8-4ebc-a275-c262d3319106-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49907290-7ea8-4ebc-a275-c262d3319106" (UID: "49907290-7ea8-4ebc-a275-c262d3319106"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:57:41.475500 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.475465 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49907290-7ea8-4ebc-a275-c262d3319106-kube-api-access-d8bsp" (OuterVolumeSpecName: "kube-api-access-d8bsp") pod "49907290-7ea8-4ebc-a275-c262d3319106" (UID: "49907290-7ea8-4ebc-a275-c262d3319106"). InnerVolumeSpecName "kube-api-access-d8bsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:57:41.574555 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.574518 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bsp\" (UniqueName: \"kubernetes.io/projected/49907290-7ea8-4ebc-a275-c262d3319106-kube-api-access-d8bsp\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:57:41.574555 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.574549 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49907290-7ea8-4ebc-a275-c262d3319106-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 19:57:41.686083 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:41.686052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:42.073577 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.073538 2579 generic.go:356] "Generic (PLEG): container finished" podID="49907290-7ea8-4ebc-a275-c262d3319106" containerID="081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e" exitCode=0 Sep 11 19:57:42.074043 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.073584 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" Sep 11 19:57:42.074043 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.073621 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" event={"ID":"49907290-7ea8-4ebc-a275-c262d3319106","Type":"ContainerDied","Data":"081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e"} Sep 11 19:57:42.074043 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.073670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x" event={"ID":"49907290-7ea8-4ebc-a275-c262d3319106","Type":"ContainerDied","Data":"e3e3ff79fb102851272f7195884420396d492fa713b1e7e4a3cda517b7899619"} Sep 11 19:57:42.074043 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.073694 2579 scope.go:117] "RemoveContainer" containerID="081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e" Sep 11 19:57:42.083503 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.083473 2579 scope.go:117] "RemoveContainer" containerID="081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e" Sep 11 19:57:42.083757 ip-10-0-138-82 kubenswrapper[2579]: E0911 19:57:42.083737 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e\": container with ID starting with 081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e not found: ID does not exist" containerID="081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e" Sep 11 19:57:42.083803 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.083765 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e"} err="failed to get container status \"081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e\": rpc error: code = NotFound desc = could not find container \"081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e\": container with ID starting with 081b869c82ef9684bb9ad8d3f09582cad4065128ae7e947136f315670063c74e not found: ID does not exist" Sep 11 19:57:42.095442 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.095413 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x"] Sep 11 19:57:42.101224 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.101204 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-105b25aa-76646cfc9d-m6p4x"] Sep 11 19:57:42.367315 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:42.367232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:43.283063 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:43.283023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:43.394861 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:43.394813 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49907290-7ea8-4ebc-a275-c262d3319106" path="/var/lib/kubelet/pods/49907290-7ea8-4ebc-a275-c262d3319106/volumes" Sep 11 19:57:44.091196 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:44.091158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:44.263490 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:44.263459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:44.771914 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:44.771882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:45.475655 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:45.475618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:46.181721 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:46.181685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:57:49.293006 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:57:49.292917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:21.848285 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:21.848256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:26.361225 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:26.361168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:26.578043 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:26.578005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:27.587705 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:27.587673 2579 scope.go:117] "RemoveContainer" containerID="cde2e61ef6bb5264e564222283640463a2a5ad30d05ced241f64607ea55afbe1" Sep 11 19:58:29.877132 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:29.877079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:32.494237 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:32.494191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:32.853244 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:32.853208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:47.056987 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:47.056947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:47.153555 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:47.153526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:47.304924 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:47.304897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:48.521612 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:48.521574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:48.617212 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:48.617167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:49.306688 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:49.306654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:49.405016 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:49.404985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:49.505303 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:49.505271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:50.709527 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:50.709495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:56.955482 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:56.955442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:57.024193 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:57.024153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:58:57.630970 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:58:57.630932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:03.799244 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:03.799208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:04.961151 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:04.961121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:07.676726 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:07.676693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:11.151859 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:11.151806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:14.477071 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:14.477031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:14.733211 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:14.733130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:27.479913 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:27.479881 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:59:27.480889 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:27.480866 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 19:59:27.615967 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:27.615938 2579 scope.go:117] "RemoveContainer" containerID="b73437e57a5d04ac0093714bf389544e9ffe3b32bcdabe43c9ad92f713041df3" Sep 11 19:59:35.593863 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:35.593810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:41.917301 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:41.917266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:42.018565 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:42.018533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:42.263426 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:42.263343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:42.594945 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:42.594914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:42.868105 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:42.868007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:45.062922 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:45.062891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:45.337739 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:45.337634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:45.450392 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:45.450349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:45.875232 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:45.875202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:46.682494 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:46.682461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:47.477317 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:47.477284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:48.208819 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:48.208786 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:49.009161 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:49.009123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:49.693607 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:49.693571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:49.764975 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:49.764339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:50.109034 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.109000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:50.212142 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.212092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:50.299096 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.299055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 19:59:50.300951 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.300929 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt"] Sep 11 19:59:50.301407 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.301393 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49907290-7ea8-4ebc-a275-c262d3319106" containerName="payload-pre-processing" Sep 11 19:59:50.301454 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.301412 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49907290-7ea8-4ebc-a275-c262d3319106" containerName="payload-pre-processing" Sep 11 19:59:50.301566 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.301541 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="49907290-7ea8-4ebc-a275-c262d3319106" containerName="payload-pre-processing" Sep 11 19:59:50.304508 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.304493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:50.307139 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.307118 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e7cdd6\"" Sep 11 19:59:50.307239 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.307214 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e7cdd6-dockercfg-zhbt4\"" Sep 11 19:59:50.315690 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.315664 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt"] Sep 11 19:59:50.393627 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.392539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pqld\" (UniqueName: \"kubernetes.io/projected/92496486-fd20-4fc7-bd43-03609f74556b-kube-api-access-9pqld\") pod \"payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt\" (UID: \"92496486-fd20-4fc7-bd43-03609f74556b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:50.394079 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.394054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92496486-fd20-4fc7-bd43-03609f74556b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt\" (UID: \"92496486-fd20-4fc7-bd43-03609f74556b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:50.495628 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.495594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqld\" (UniqueName: \"kubernetes.io/projected/92496486-fd20-4fc7-bd43-03609f74556b-kube-api-access-9pqld\") pod \"payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt\" (UID: \"92496486-fd20-4fc7-bd43-03609f74556b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:50.495812 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.495698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92496486-fd20-4fc7-bd43-03609f74556b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt\" (UID: \"92496486-fd20-4fc7-bd43-03609f74556b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:50.496384 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.496366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92496486-fd20-4fc7-bd43-03609f74556b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt\" (UID: \"92496486-fd20-4fc7-bd43-03609f74556b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:50.504175 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.504155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqld\" (UniqueName: \"kubernetes.io/projected/92496486-fd20-4fc7-bd43-03609f74556b-kube-api-access-9pqld\") pod \"payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt\" (UID: \"92496486-fd20-4fc7-bd43-03609f74556b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:50.616589 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.616542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:50.891316 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:50.891287 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt"] Sep 11 19:59:50.893207 ip-10-0-138-82 kubenswrapper[2579]: W0911 19:59:50.893178 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92496486_fd20_4fc7_bd43_03609f74556b.slice/crio-67056155daeb8deea503fcb42c9aacd48c3abe6c614a9f17f94910c75db94445 WatchSource:0}: Error finding container 67056155daeb8deea503fcb42c9aacd48c3abe6c614a9f17f94910c75db94445: Status 404 returned error can't find the container with id 67056155daeb8deea503fcb42c9aacd48c3abe6c614a9f17f94910c75db94445 Sep 11 19:59:51.870862 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:51.870802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" event={"ID":"92496486-fd20-4fc7-bd43-03609f74556b","Type":"ContainerStarted","Data":"74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340"} Sep 11 19:59:51.871066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:51.870879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" event={"ID":"92496486-fd20-4fc7-bd43-03609f74556b","Type":"ContainerStarted","Data":"67056155daeb8deea503fcb42c9aacd48c3abe6c614a9f17f94910c75db94445"} Sep 11 19:59:51.871066 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:51.870901 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 19:59:51.890599 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:51.890546 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" podStartSLOduration=1.890527027 podStartE2EDuration="1.890527027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:59:51.888123641 +0000 UTC m=+1825.021436231" watchObservedRunningTime="2026-09-11 19:59:51.890527027 +0000 UTC m=+1825.023839616" Sep 11 19:59:52.444976 ip-10-0-138-82 kubenswrapper[2579]: I0911 19:59:52.444946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:00.139999 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.139963 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t"] Sep 11 20:00:00.144574 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.144552 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz"] Sep 11 20:00:00.144715 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.144672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" Sep 11 20:00:00.147758 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.147737 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 20:00:00.148137 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.148121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" Sep 11 20:00:00.157424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.157401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" Sep 11 20:00:00.160984 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.160494 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t"] Sep 11 20:00:00.164512 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.164481 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz"] Sep 11 20:00:00.173028 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.173003 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" Sep 11 20:00:00.178131 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.178039 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p"] Sep 11 20:00:00.182749 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.182725 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" Sep 11 20:00:00.189952 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.189571 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj"] Sep 11 20:00:00.197767 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.197723 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p"] Sep 11 20:00:00.197965 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.197891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj" Sep 11 20:00:00.198794 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.198725 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj"] Sep 11 20:00:00.228809 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.228786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" Sep 11 20:00:00.241768 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.241730 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj" Sep 11 20:00:00.486254 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.485592 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz"] Sep 11 20:00:00.487152 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:00:00.487127 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf339cb7_d4d2_4d55_b124_d1d0917a4a97.slice/crio-5b7ebb7e4e8c25ca9e5c7f4504e459a9ff2a766c7329e2a49bde96fb1dcd13e5 WatchSource:0}: Error finding container 5b7ebb7e4e8c25ca9e5c7f4504e459a9ff2a766c7329e2a49bde96fb1dcd13e5: Status 404 returned error can't find the container with id 5b7ebb7e4e8c25ca9e5c7f4504e459a9ff2a766c7329e2a49bde96fb1dcd13e5 Sep 11 20:00:00.489098 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.489073 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t"] Sep 11 20:00:00.490466 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:00:00.490443 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18b0636_edef_4040_afac_89b31fd29673.slice/crio-8dccf0bc21061d5727bcde4f60b53d7f35f724a48548ff331573924629a52b59 WatchSource:0}: Error finding container 8dccf0bc21061d5727bcde4f60b53d7f35f724a48548ff331573924629a52b59: Status 404 returned error can't find the container with id 8dccf0bc21061d5727bcde4f60b53d7f35f724a48548ff331573924629a52b59 Sep 11 20:00:00.539805 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.539778 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p"] Sep 11 20:00:00.541901 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:00:00.541872 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode679464e_5afa_430d_9f26_d72f042dacbc.slice/crio-661270fd1409d0352de5fca5142e040d9d7c4fc85eda3d28242a5174ecc69bbd WatchSource:0}: Error finding container 661270fd1409d0352de5fca5142e040d9d7c4fc85eda3d28242a5174ecc69bbd: Status 404 returned error can't find the container with id 661270fd1409d0352de5fca5142e040d9d7c4fc85eda3d28242a5174ecc69bbd Sep 11 20:00:00.550373 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.550351 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj"] Sep 11 20:00:00.551645 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:00:00.551619 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f21ccb_3d71_46bf_b4a5_abe059193890.slice/crio-0c7988b4c9a7b5aff7d5f2a1cbf3bc20f89c7308c8c8c94f468e860140f7aebf WatchSource:0}: Error finding container 0c7988b4c9a7b5aff7d5f2a1cbf3bc20f89c7308c8c8c94f468e860140f7aebf: Status 404 returned error can't find the container with id 0c7988b4c9a7b5aff7d5f2a1cbf3bc20f89c7308c8c8c94f468e860140f7aebf Sep 11 20:00:00.927068 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.927028 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" event={"ID":"a18b0636-edef-4040-afac-89b31fd29673","Type":"ContainerStarted","Data":"c8880a701c54358e1f9e7b4ccc1b32053f76a5c39c49b712ea60b27206f0a0ed"} Sep 11 20:00:00.927245 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.927083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" event={"ID":"a18b0636-edef-4040-afac-89b31fd29673","Type":"ContainerStarted","Data":"8dccf0bc21061d5727bcde4f60b53d7f35f724a48548ff331573924629a52b59"} Sep 11 20:00:00.928346 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.928322 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj" event={"ID":"70f21ccb-3d71-46bf-b4a5-abe059193890","Type":"ContainerStarted","Data":"0c7988b4c9a7b5aff7d5f2a1cbf3bc20f89c7308c8c8c94f468e860140f7aebf"} Sep 11 20:00:00.929907 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.929889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" event={"ID":"e679464e-5afa-430d-9f26-d72f042dacbc","Type":"ContainerStarted","Data":"1a7232b56683eb742197dd3401b066b916305d453b54b1992a8299f480f4517e"} Sep 11 20:00:00.929975 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.929923 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" event={"ID":"e679464e-5afa-430d-9f26-d72f042dacbc","Type":"ContainerStarted","Data":"661270fd1409d0352de5fca5142e040d9d7c4fc85eda3d28242a5174ecc69bbd"} Sep 11 20:00:00.931464 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.931446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" event={"ID":"df339cb7-d4d2-4d55-b124-d1d0917a4a97","Type":"ContainerStarted","Data":"4f031ec3f0440b63736998689b468bc588328e35a22f2a1ff731bca72c2875fe"} Sep 11 20:00:00.931545 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.931476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" event={"ID":"df339cb7-d4d2-4d55-b124-d1d0917a4a97","Type":"ContainerStarted","Data":"5b7ebb7e4e8c25ca9e5c7f4504e459a9ff2a766c7329e2a49bde96fb1dcd13e5"} Sep 11 20:00:00.943480 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.943432 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" podStartSLOduration=0.943420363 podStartE2EDuration="943.420363ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:00.942149627 +0000 UTC m=+1834.075462217" watchObservedRunningTime="2026-09-11 20:00:00.943420363 +0000 UTC m=+1834.076732952" Sep 11 20:00:00.957169 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.957119 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" podStartSLOduration=0.957104348 podStartE2EDuration="957.104348ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:00.955751015 +0000 UTC m=+1834.089063607" watchObservedRunningTime="2026-09-11 20:00:00.957104348 +0000 UTC m=+1834.090416937" Sep 11 20:00:00.970027 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:00.969976 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" podStartSLOduration=0.969961801 podStartE2EDuration="969.961801ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:00.969251543 +0000 UTC m=+1834.102564133" watchObservedRunningTime="2026-09-11 20:00:00.969961801 +0000 UTC m=+1834.103274440" Sep 11 20:00:01.504093 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.504059 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:01.939532 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.939492 2579 generic.go:356] "Generic (PLEG): container finished" podID="df339cb7-d4d2-4d55-b124-d1d0917a4a97" containerID="4f031ec3f0440b63736998689b468bc588328e35a22f2a1ff731bca72c2875fe" exitCode=0 Sep 11 20:00:01.939720 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.939583 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" event={"ID":"df339cb7-d4d2-4d55-b124-d1d0917a4a97","Type":"ContainerDied","Data":"4f031ec3f0440b63736998689b468bc588328e35a22f2a1ff731bca72c2875fe"} Sep 11 20:00:01.941202 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.941173 2579 generic.go:356] "Generic (PLEG): container finished" podID="a18b0636-edef-4040-afac-89b31fd29673" containerID="c8880a701c54358e1f9e7b4ccc1b32053f76a5c39c49b712ea60b27206f0a0ed" exitCode=0 Sep 11 20:00:01.941302 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.941251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" event={"ID":"a18b0636-edef-4040-afac-89b31fd29673","Type":"ContainerDied","Data":"c8880a701c54358e1f9e7b4ccc1b32053f76a5c39c49b712ea60b27206f0a0ed"} Sep 11 20:00:01.942691 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.942671 2579 generic.go:356] "Generic (PLEG): container finished" podID="70f21ccb-3d71-46bf-b4a5-abe059193890" containerID="6b08b91b94b8981be1b4178ab105412ea0f7c5bdd304de2d410baa294e025624" exitCode=0 Sep 11 20:00:01.942789 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.942730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj" event={"ID":"70f21ccb-3d71-46bf-b4a5-abe059193890","Type":"ContainerDied","Data":"6b08b91b94b8981be1b4178ab105412ea0f7c5bdd304de2d410baa294e025624"} Sep 11 20:00:01.944339 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.944320 2579 generic.go:356] "Generic (PLEG): container finished" podID="e679464e-5afa-430d-9f26-d72f042dacbc" containerID="1a7232b56683eb742197dd3401b066b916305d453b54b1992a8299f480f4517e" exitCode=0 Sep 11 20:00:01.944436 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:01.944395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" event={"ID":"e679464e-5afa-430d-9f26-d72f042dacbc","Type":"ContainerDied","Data":"1a7232b56683eb742197dd3401b066b916305d453b54b1992a8299f480f4517e"} Sep 11 20:00:02.199946 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:02.199856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:02.882027 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:02.881994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 20:00:03.043544 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.043040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:03.159787 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.159707 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" Sep 11 20:00:03.166222 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.166198 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" Sep 11 20:00:03.228319 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.228294 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj" Sep 11 20:00:03.231764 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.231735 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" Sep 11 20:00:03.564048 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.563985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:03.958611 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.958526 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" Sep 11 20:00:03.958611 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.958544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t" event={"ID":"a18b0636-edef-4040-afac-89b31fd29673","Type":"ContainerDied","Data":"8dccf0bc21061d5727bcde4f60b53d7f35f724a48548ff331573924629a52b59"} Sep 11 20:00:03.958611 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.958591 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dccf0bc21061d5727bcde4f60b53d7f35f724a48548ff331573924629a52b59" Sep 11 20:00:03.960130 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.960106 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj" Sep 11 20:00:03.960130 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.960118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-d67pj" event={"ID":"70f21ccb-3d71-46bf-b4a5-abe059193890","Type":"ContainerDied","Data":"0c7988b4c9a7b5aff7d5f2a1cbf3bc20f89c7308c8c8c94f468e860140f7aebf"} Sep 11 20:00:03.960349 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.960161 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7988b4c9a7b5aff7d5f2a1cbf3bc20f89c7308c8c8c94f468e860140f7aebf" Sep 11 20:00:03.961975 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.961954 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" Sep 11 20:00:03.961975 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.961965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p" event={"ID":"e679464e-5afa-430d-9f26-d72f042dacbc","Type":"ContainerDied","Data":"661270fd1409d0352de5fca5142e040d9d7c4fc85eda3d28242a5174ecc69bbd"} Sep 11 20:00:03.962163 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.962004 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="661270fd1409d0352de5fca5142e040d9d7c4fc85eda3d28242a5174ecc69bbd" Sep 11 20:00:03.963655 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.963630 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" Sep 11 20:00:03.963655 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.963638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz" event={"ID":"df339cb7-d4d2-4d55-b124-d1d0917a4a97","Type":"ContainerDied","Data":"5b7ebb7e4e8c25ca9e5c7f4504e459a9ff2a766c7329e2a49bde96fb1dcd13e5"} Sep 11 20:00:03.963783 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:03.963681 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b7ebb7e4e8c25ca9e5c7f4504e459a9ff2a766c7329e2a49bde96fb1dcd13e5" Sep 11 20:00:04.310481 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:04.310450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:04.981635 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:04.981600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:05.697713 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:05.697683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:06.362568 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:06.362530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:06.577294 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:06.577258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:07.104721 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:07.104691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:07.952996 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:07.952964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:08.586664 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:08.586631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:09.245508 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:09.245467 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:09.998636 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:09.998598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:10.223614 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.223575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:10.911274 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911241 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv"] Sep 11 20:00:10.911763 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911748 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f21ccb-3d71-46bf-b4a5-abe059193890" containerName="cleanup" Sep 11 20:00:10.911763 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911763 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f21ccb-3d71-46bf-b4a5-abe059193890" containerName="cleanup" Sep 11 20:00:10.911905 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911781 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e679464e-5afa-430d-9f26-d72f042dacbc" containerName="cleanup" Sep 11 20:00:10.911905 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911787 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e679464e-5afa-430d-9f26-d72f042dacbc" containerName="cleanup" Sep 11 20:00:10.911905 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911808 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18b0636-edef-4040-afac-89b31fd29673" containerName="cleanup" Sep 11 20:00:10.911905 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911814 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18b0636-edef-4040-afac-89b31fd29673" containerName="cleanup" Sep 11 20:00:10.911905 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911834 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df339cb7-d4d2-4d55-b124-d1d0917a4a97" containerName="cleanup" Sep 11 20:00:10.911905 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911839 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df339cb7-d4d2-4d55-b124-d1d0917a4a97" containerName="cleanup" Sep 11 20:00:10.911905 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911903 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e679464e-5afa-430d-9f26-d72f042dacbc" containerName="cleanup" Sep 11 20:00:10.912268 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911913 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18b0636-edef-4040-afac-89b31fd29673" containerName="cleanup" Sep 11 20:00:10.912268 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911919 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f21ccb-3d71-46bf-b4a5-abe059193890" containerName="cleanup" Sep 11 20:00:10.912268 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.911929 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df339cb7-d4d2-4d55-b124-d1d0917a4a97" containerName="cleanup" Sep 11 20:00:10.915326 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.915308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:10.918022 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.917998 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b1a074-openshift-default-dockercfg-9qrls\"" Sep 11 20:00:10.928676 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:10.928619 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv"] Sep 11 20:00:11.019209 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-podinfo\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.019209 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-credential-socket\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.019490 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cv5q\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-kube-api-access-4cv5q\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.019490 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-data\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.019490 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-token\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.019490 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-envoy\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.019490 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-certs\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.019490 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-socket\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.019794 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.019563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25418406-96a6-4eec-a5a4-98e672c5dcb5-istiod-ca-cert\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.120756 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.120715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25418406-96a6-4eec-a5a4-98e672c5dcb5-istiod-ca-cert\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.120980 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.120780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-podinfo\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.120980 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.120818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-credential-socket\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.120980 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.120896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cv5q\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-kube-api-access-4cv5q\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.120980 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.120930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-data\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.120980 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.120956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-token\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.121255 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.120997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-envoy\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.121255 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.121033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-certs\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.121255 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.121078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-socket\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.121597 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.121564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25418406-96a6-4eec-a5a4-98e672c5dcb5-istiod-ca-cert\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.121704 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.121613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-data\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.121704 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.121647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-credential-socket\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.121849 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.121802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-certs\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.121919 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.121850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-socket\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.123914 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.123885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-podinfo\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.123914 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.123907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-envoy\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.129932 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.129907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-token\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.130017 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.130005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cv5q\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-kube-api-access-4cv5q\") pod \"e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.228956 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.228858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:11.474264 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.474227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:11.523206 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.523183 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv"] Sep 11 20:00:11.525719 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:00:11.525682 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25418406_96a6_4eec_a5a4_98e672c5dcb5.slice/crio-f67094af3551341b1dc2f5e33fb3b2417dcf8fd64a0592460e5fe17a59913f46 WatchSource:0}: Error finding container f67094af3551341b1dc2f5e33fb3b2417dcf8fd64a0592460e5fe17a59913f46: Status 404 returned error can't find the container with id f67094af3551341b1dc2f5e33fb3b2417dcf8fd64a0592460e5fe17a59913f46 Sep 11 20:00:11.528128 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.528089 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:00:11.528203 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.528163 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:00:11.528203 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.528192 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:00:11.532485 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:11.532456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:12.026703 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:12.026664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" event={"ID":"25418406-96a6-4eec-a5a4-98e672c5dcb5","Type":"ContainerStarted","Data":"be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb"} Sep 11 20:00:12.027117 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:12.026722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" event={"ID":"25418406-96a6-4eec-a5a4-98e672c5dcb5","Type":"ContainerStarted","Data":"f67094af3551341b1dc2f5e33fb3b2417dcf8fd64a0592460e5fe17a59913f46"} Sep 11 20:00:12.047033 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:12.046984 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" podStartSLOduration=2.046967221 podStartE2EDuration="2.046967221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:12.045181113 +0000 UTC m=+1845.178493728" watchObservedRunningTime="2026-09-11 20:00:12.046967221 +0000 UTC m=+1845.180279809" Sep 11 20:00:12.229636 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:12.229597 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:12.235245 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:12.235220 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:13.033637 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:13.033607 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:13.034680 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:13.034662 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:13.180677 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:13.180632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:13.920882 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:13.920800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:14.037323 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:14.037197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:14.708590 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:14.708542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:15.275431 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.275405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:15.301881 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.301848 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r"] Sep 11 20:00:15.302152 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.302120 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" podUID="f471704e-af44-400e-bb99-cdff6e6fbea7" containerName="payload-processing" containerID="cri-o://786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00" gracePeriod=30 Sep 11 20:00:15.375322 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.375285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:15.477912 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.477884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:15.560912 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.560885 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 20:00:15.578249 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.578219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:15.682442 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.681305 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bd6m\" (UniqueName: \"kubernetes.io/projected/f471704e-af44-400e-bb99-cdff6e6fbea7-kube-api-access-8bd6m\") pod \"f471704e-af44-400e-bb99-cdff6e6fbea7\" (UID: \"f471704e-af44-400e-bb99-cdff6e6fbea7\") " Sep 11 20:00:15.682442 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.681410 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f471704e-af44-400e-bb99-cdff6e6fbea7-plugins-config-volume\") pod \"f471704e-af44-400e-bb99-cdff6e6fbea7\" (UID: \"f471704e-af44-400e-bb99-cdff6e6fbea7\") " Sep 11 20:00:15.682442 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.682152 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f471704e-af44-400e-bb99-cdff6e6fbea7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f471704e-af44-400e-bb99-cdff6e6fbea7" (UID: "f471704e-af44-400e-bb99-cdff6e6fbea7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:00:15.685804 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.685770 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f471704e-af44-400e-bb99-cdff6e6fbea7-kube-api-access-8bd6m" (OuterVolumeSpecName: "kube-api-access-8bd6m") pod "f471704e-af44-400e-bb99-cdff6e6fbea7" (UID: "f471704e-af44-400e-bb99-cdff6e6fbea7"). InnerVolumeSpecName "kube-api-access-8bd6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:00:15.748921 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.748884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:15.783430 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.783398 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bd6m\" (UniqueName: \"kubernetes.io/projected/f471704e-af44-400e-bb99-cdff6e6fbea7-kube-api-access-8bd6m\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:00:15.783430 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.783428 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f471704e-af44-400e-bb99-cdff6e6fbea7-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:00:15.998721 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:15.998635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:16.056195 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.056162 2579 generic.go:356] "Generic (PLEG): container finished" podID="f471704e-af44-400e-bb99-cdff6e6fbea7" containerID="786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00" exitCode=0 Sep 11 20:00:16.056383 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.056207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" event={"ID":"f471704e-af44-400e-bb99-cdff6e6fbea7","Type":"ContainerDied","Data":"786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00"} Sep 11 20:00:16.056383 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.056214 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" Sep 11 20:00:16.056383 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.056250 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r" event={"ID":"f471704e-af44-400e-bb99-cdff6e6fbea7","Type":"ContainerDied","Data":"312b1247b66ec6f0d0c9cd71a0b62c350748e327f19434b4bf10187002c5cc76"} Sep 11 20:00:16.056383 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.056272 2579 scope.go:117] "RemoveContainer" containerID="786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00" Sep 11 20:00:16.076568 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.076544 2579 scope.go:117] "RemoveContainer" containerID="786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00" Sep 11 20:00:16.076963 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:00:16.076937 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00\": container with ID starting with 786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00 not found: ID does not exist" containerID="786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00" Sep 11 20:00:16.077023 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.076977 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00"} err="failed to get container status \"786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00\": rpc error: code = NotFound desc = could not find container \"786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00\": container with ID starting with 786c27b894940c9244831050f996ccd88a08151aad3e91674702bf2c864ddd00 not found: ID does not exist" Sep 11 20:00:16.087344 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.087305 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r"] Sep 11 20:00:16.091634 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.091609 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b032e-76485b7f59-5cr6r"] Sep 11 20:00:16.572609 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:16.572582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:17.284343 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:17.284305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:17.395510 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:17.395480 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f471704e-af44-400e-bb99-cdff6e6fbea7" path="/var/lib/kubelet/pods/f471704e-af44-400e-bb99-cdff6e6fbea7/volumes" Sep 11 20:00:17.916882 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:17.916853 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:18.686281 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:18.686244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:19.360478 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:19.360447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:19.461728 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:19.461680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:19.805519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:19.805485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:20.630752 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:20.630716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:20.728004 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:20.727973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:21.167101 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:21.167072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:21.895718 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:21.895695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:22.573932 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:22.573781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:23.448957 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:23.448916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:23.549738 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:23.549688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:27.054727 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:27.054692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:27.638435 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:27.638400 2579 scope.go:117] "RemoveContainer" containerID="425b9f2cc5b5a2347569eb1e04c3368c54268d1b14987b5929be81bdb839fde7" Sep 11 20:00:27.647811 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:27.647789 2579 scope.go:117] "RemoveContainer" containerID="21e8214bdbccedff34df269b1a75bb5fe07558264480908152353e505d8ed448" Sep 11 20:00:40.279425 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:40.279385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:41.493320 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:41.493293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:42.139643 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:42.139611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:42.800670 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:42.800639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:43.450078 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:43.450039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:43.938609 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:43.938572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:44.038903 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:44.038865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:44.139063 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:44.139018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:44.242016 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:44.241917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:44.835349 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:44.835319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:44.933736 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:44.933711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:45.779095 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:45.779057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:46.436210 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:46.436168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:47.292002 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:47.291967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:47.982263 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:47.982225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:48.795409 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:48.795322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:49.499544 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:49.499510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:50.379755 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:50.379712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:51.003733 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:51.003691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:51.614772 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:51.614738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:52.375978 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:52.375875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:55.317230 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:55.317197 2579 generic.go:356] "Generic (PLEG): container finished" podID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerID="be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb" exitCode=0 Sep 11 20:00:55.317646 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:55.317275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" event={"ID":"25418406-96a6-4eec-a5a4-98e672c5dcb5","Type":"ContainerDied","Data":"be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb"} Sep 11 20:00:55.317756 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:55.317741 2579 scope.go:117] "RemoveContainer" containerID="be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb" Sep 11 20:00:55.319032 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:55.318970 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:00:55.319121 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:55.319072 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:00:55.319121 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:55.319103 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:00:56.325360 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:56.325321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" event={"ID":"25418406-96a6-4eec-a5a4-98e672c5dcb5","Type":"ContainerStarted","Data":"e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6"} Sep 11 20:00:56.877576 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:56.877505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:00:57.230075 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:57.229983 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:57.231361 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:57.231339 2579 patch_prober.go:28] interesting pod/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 11 20:00:57.231424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:57.231388 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 11 20:00:58.034041 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:58.034005 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:00:58.229750 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:58.229701 2579 patch_prober.go:28] interesting pod/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 11 20:00:58.229996 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:58.229777 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 11 20:00:59.229452 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:59.229416 2579 patch_prober.go:28] interesting pod/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 11 20:00:59.229901 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:59.229475 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 11 20:00:59.824622 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:00:59.824572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:00.229376 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:00.229276 2579 patch_prober.go:28] interesting pod/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 11 20:01:00.229376 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:00.229343 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 11 20:01:01.229705 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:01.229661 2579 patch_prober.go:28] interesting pod/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 11 20:01:01.230143 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:01.229724 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 11 20:01:02.233479 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.233442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:01:02.234398 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.234378 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:01:02.448217 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.448186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:02.479745 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.479707 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt"] Sep 11 20:01:02.479998 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.479974 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" podUID="92496486-fd20-4fc7-bd43-03609f74556b" containerName="payload-processing" containerID="cri-o://74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340" gracePeriod=30 Sep 11 20:01:02.547284 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.547253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:02.649230 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.649191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:02.713517 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.713489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:02.813010 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.812985 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 20:01:02.846746 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.846710 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pqld\" (UniqueName: \"kubernetes.io/projected/92496486-fd20-4fc7-bd43-03609f74556b-kube-api-access-9pqld\") pod \"92496486-fd20-4fc7-bd43-03609f74556b\" (UID: \"92496486-fd20-4fc7-bd43-03609f74556b\") " Sep 11 20:01:02.846982 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.846943 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92496486-fd20-4fc7-bd43-03609f74556b-plugins-config-volume\") pod \"92496486-fd20-4fc7-bd43-03609f74556b\" (UID: \"92496486-fd20-4fc7-bd43-03609f74556b\") " Sep 11 20:01:02.847319 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.847290 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92496486-fd20-4fc7-bd43-03609f74556b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92496486-fd20-4fc7-bd43-03609f74556b" (UID: "92496486-fd20-4fc7-bd43-03609f74556b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:01:02.849119 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.849085 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92496486-fd20-4fc7-bd43-03609f74556b-kube-api-access-9pqld" (OuterVolumeSpecName: "kube-api-access-9pqld") pod "92496486-fd20-4fc7-bd43-03609f74556b" (UID: "92496486-fd20-4fc7-bd43-03609f74556b"). InnerVolumeSpecName "kube-api-access-9pqld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:02.948407 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.948363 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92496486-fd20-4fc7-bd43-03609f74556b-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:02.948407 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.948405 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pqld\" (UniqueName: \"kubernetes.io/projected/92496486-fd20-4fc7-bd43-03609f74556b-kube-api-access-9pqld\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:02.999276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:02.999241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:03.080321 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.080282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:03.377576 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.377486 2579 generic.go:356] "Generic (PLEG): container finished" podID="92496486-fd20-4fc7-bd43-03609f74556b" containerID="74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340" exitCode=0 Sep 11 20:01:03.377576 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.377548 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" Sep 11 20:01:03.378138 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.377582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" event={"ID":"92496486-fd20-4fc7-bd43-03609f74556b","Type":"ContainerDied","Data":"74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340"} Sep 11 20:01:03.378138 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.377677 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt" event={"ID":"92496486-fd20-4fc7-bd43-03609f74556b","Type":"ContainerDied","Data":"67056155daeb8deea503fcb42c9aacd48c3abe6c614a9f17f94910c75db94445"} Sep 11 20:01:03.378138 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.377712 2579 scope.go:117] "RemoveContainer" containerID="74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340" Sep 11 20:01:03.387652 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.387626 2579 scope.go:117] "RemoveContainer" containerID="74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340" Sep 11 20:01:03.388009 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:01:03.387988 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340\": container with ID starting with 74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340 not found: ID does not exist" containerID="74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340" Sep 11 20:01:03.388084 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.388018 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340"} err="failed to get container status \"74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340\": rpc error: code = NotFound desc = could not find container \"74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340\": container with ID starting with 74d73f530b5a10ec4b95d7e45ea7a22dccaed1491265555f8af7db7d32f2c340 not found: ID does not exist" Sep 11 20:01:03.399847 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.399804 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt"] Sep 11 20:01:03.406124 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.406095 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e7cdd6-88bc89bfc-r95tt"] Sep 11 20:01:03.792763 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:03.792733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:04.636499 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:04.636458 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:05.393802 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:05.393766 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92496486-fd20-4fc7-bd43-03609f74556b" path="/var/lib/kubelet/pods/92496486-fd20-4fc7-bd43-03609f74556b/volumes" Sep 11 20:01:05.523469 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:05.523438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:06.235266 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:06.235233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:06.999280 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:06.999246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:07.673494 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:07.673462 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:08.871063 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:08.871023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:08.982009 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:08.981959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:09.208344 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:09.208245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:09.778524 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:09.778495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:10.472571 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:10.472538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:11.384185 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:11.384140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:12.052624 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:12.052582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:12.679594 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:12.679561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:13.378964 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:13.378933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:14.013081 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:14.013050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:14.808070 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:14.808037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:17.248471 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:17.248431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:17.687842 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:17.687800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:18.366681 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:18.366654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:19.110755 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:19.110723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:19.763806 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:19.763774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:20.487712 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:20.487679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:20.902271 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:20.902240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:21.004088 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:21.004056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:21.166031 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:21.165903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:21.871914 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:21.871882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:22.326516 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:22.326461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:25.313988 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:25.313957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:25.381722 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:25.381675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:25.444404 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:25.444368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:25.507426 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:25.507376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:25.848188 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:25.848126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:25.948234 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:25.948185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:26.594380 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:26.594337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:27.399048 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:27.398990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:28.170473 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:28.170430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:29.063901 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:29.063863 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:31.616531 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:31.616480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:31.690026 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:31.689993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:31.944597 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:31.944495 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv"] Sep 11 20:01:31.944777 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:31.944753 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" containerID="cri-o://e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6" gracePeriod=30 Sep 11 20:01:31.962448 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:31.962411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:32.236651 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:32.236569 2579 patch_prober.go:28] interesting pod/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 20:01:32.236651 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:32.236631 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 20:01:33.099731 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.099691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:33.706622 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.706590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:33.708116 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.708094 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t"] Sep 11 20:01:33.711440 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.711418 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a8ee95-29819280-78h2t"] Sep 11 20:01:33.741626 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.741571 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk"] Sep 11 20:01:33.742018 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.741974 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" podUID="371ac68d-4503-4374-b598-033d7687e0d1" containerName="payload-processing" containerID="cri-o://3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e" gracePeriod=30 Sep 11 20:01:33.745854 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.745811 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s"] Sep 11 20:01:33.746335 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.746308 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" podUID="146408ba-5617-47ee-adba-3baf420d38d1" containerName="payload-pre-processing" containerID="cri-o://20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d" gracePeriod=30 Sep 11 20:01:33.785733 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.785688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:33.848178 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.848145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:33.890869 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.890808 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" podUID="146408ba-5617-47ee-adba-3baf420d38d1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.38:9004: connect: connection refused" Sep 11 20:01:33.891065 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.890879 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" podUID="371ac68d-4503-4374-b598-033d7687e0d1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.37:9004: connect: connection refused" Sep 11 20:01:33.949120 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:33.949092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:34.082956 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.082909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:34.099184 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.099157 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 20:01:34.116002 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.115965 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 20:01:34.207758 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.207724 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ac68d-4503-4374-b598-033d7687e0d1-plugins-config-volume\") pod \"371ac68d-4503-4374-b598-033d7687e0d1\" (UID: \"371ac68d-4503-4374-b598-033d7687e0d1\") " Sep 11 20:01:34.207983 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.207771 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxmxf\" (UniqueName: \"kubernetes.io/projected/371ac68d-4503-4374-b598-033d7687e0d1-kube-api-access-mxmxf\") pod \"371ac68d-4503-4374-b598-033d7687e0d1\" (UID: \"371ac68d-4503-4374-b598-033d7687e0d1\") " Sep 11 20:01:34.207983 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.207793 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146408ba-5617-47ee-adba-3baf420d38d1-plugins-config-volume\") pod \"146408ba-5617-47ee-adba-3baf420d38d1\" (UID: \"146408ba-5617-47ee-adba-3baf420d38d1\") " Sep 11 20:01:34.207983 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.207871 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27594\" (UniqueName: \"kubernetes.io/projected/146408ba-5617-47ee-adba-3baf420d38d1-kube-api-access-27594\") pod \"146408ba-5617-47ee-adba-3baf420d38d1\" (UID: \"146408ba-5617-47ee-adba-3baf420d38d1\") " Sep 11 20:01:34.208235 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.208207 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371ac68d-4503-4374-b598-033d7687e0d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "371ac68d-4503-4374-b598-033d7687e0d1" (UID: "371ac68d-4503-4374-b598-033d7687e0d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:01:34.208320 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.208210 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146408ba-5617-47ee-adba-3baf420d38d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "146408ba-5617-47ee-adba-3baf420d38d1" (UID: "146408ba-5617-47ee-adba-3baf420d38d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:01:34.208377 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.208358 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ac68d-4503-4374-b598-033d7687e0d1-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:34.208422 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.208381 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146408ba-5617-47ee-adba-3baf420d38d1-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:34.210839 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.210794 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371ac68d-4503-4374-b598-033d7687e0d1-kube-api-access-mxmxf" (OuterVolumeSpecName: "kube-api-access-mxmxf") pod "371ac68d-4503-4374-b598-033d7687e0d1" (UID: "371ac68d-4503-4374-b598-033d7687e0d1"). InnerVolumeSpecName "kube-api-access-mxmxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:34.210959 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.210868 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146408ba-5617-47ee-adba-3baf420d38d1-kube-api-access-27594" (OuterVolumeSpecName: "kube-api-access-27594") pod "146408ba-5617-47ee-adba-3baf420d38d1" (UID: "146408ba-5617-47ee-adba-3baf420d38d1"). InnerVolumeSpecName "kube-api-access-27594". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:34.309531 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.309484 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxmxf\" (UniqueName: \"kubernetes.io/projected/371ac68d-4503-4374-b598-033d7687e0d1-kube-api-access-mxmxf\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:34.309531 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.309521 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27594\" (UniqueName: \"kubernetes.io/projected/146408ba-5617-47ee-adba-3baf420d38d1-kube-api-access-27594\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:34.375938 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.375899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:34.572356 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.572267 2579 generic.go:356] "Generic (PLEG): container finished" podID="371ac68d-4503-4374-b598-033d7687e0d1" containerID="3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e" exitCode=0 Sep 11 20:01:34.572356 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.572313 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" Sep 11 20:01:34.572601 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.572354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" event={"ID":"371ac68d-4503-4374-b598-033d7687e0d1","Type":"ContainerDied","Data":"3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e"} Sep 11 20:01:34.572601 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.572406 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk" event={"ID":"371ac68d-4503-4374-b598-033d7687e0d1","Type":"ContainerDied","Data":"b39685f71e61ed99bfcc27368cbe0451292730f9d3caf1c4ae1c95cf8e2d8d38"} Sep 11 20:01:34.572601 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.572430 2579 scope.go:117] "RemoveContainer" containerID="3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e" Sep 11 20:01:34.573774 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.573754 2579 generic.go:356] "Generic (PLEG): container finished" podID="146408ba-5617-47ee-adba-3baf420d38d1" containerID="20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d" exitCode=0 Sep 11 20:01:34.573894 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.573795 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" Sep 11 20:01:34.573894 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.573854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" event={"ID":"146408ba-5617-47ee-adba-3baf420d38d1","Type":"ContainerDied","Data":"20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d"} Sep 11 20:01:34.574017 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.573901 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s" event={"ID":"146408ba-5617-47ee-adba-3baf420d38d1","Type":"ContainerDied","Data":"f97283886265087f089753ec20f7816159eb2437ad585e7191dabc8300097c90"} Sep 11 20:01:34.583407 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.583379 2579 scope.go:117] "RemoveContainer" containerID="3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e" Sep 11 20:01:34.583753 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:01:34.583732 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e\": container with ID starting with 3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e not found: ID does not exist" containerID="3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e" Sep 11 20:01:34.583865 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.583763 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e"} err="failed to get container status \"3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e\": rpc error: code = NotFound desc = could not find container \"3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e\": container with ID starting with 3300512028ecdfa6bf113434f6faec1f96df7e4a1159939c1eef761c5d1a268e not found: ID does not exist" Sep 11 20:01:34.583865 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.583780 2579 scope.go:117] "RemoveContainer" containerID="20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d" Sep 11 20:01:34.592385 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.592355 2579 scope.go:117] "RemoveContainer" containerID="20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d" Sep 11 20:01:34.592636 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:01:34.592618 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d\": container with ID starting with 20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d not found: ID does not exist" containerID="20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d" Sep 11 20:01:34.592694 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.592645 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d"} err="failed to get container status \"20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d\": rpc error: code = NotFound desc = could not find container \"20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d\": container with ID starting with 20d626dfbdf6fe04b581fe61bbadd15f299d99cf0a6201f0d25fd1d4fc01e51d not found: ID does not exist" Sep 11 20:01:34.599962 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.599937 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk"] Sep 11 20:01:34.605009 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.604981 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a8ee95-58774d757c-5b4rk"] Sep 11 20:01:34.616784 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.616757 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s"] Sep 11 20:01:34.621508 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.621480 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a8ee95-6b84556ff4-gp78s"] Sep 11 20:01:34.989113 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:34.989029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:35.395714 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:35.395680 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146408ba-5617-47ee-adba-3baf420d38d1" path="/var/lib/kubelet/pods/146408ba-5617-47ee-adba-3baf420d38d1/volumes" Sep 11 20:01:35.396130 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:35.396115 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371ac68d-4503-4374-b598-033d7687e0d1" path="/var/lib/kubelet/pods/371ac68d-4503-4374-b598-033d7687e0d1/volumes" Sep 11 20:01:35.396378 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:35.396367 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18b0636-edef-4040-afac-89b31fd29673" path="/var/lib/kubelet/pods/a18b0636-edef-4040-afac-89b31fd29673/volumes" Sep 11 20:01:35.597142 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:35.597110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:36.274411 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:36.274381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:37.143784 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.143746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:37.217814 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.217745 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:01:37.243268 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-podinfo\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243445 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243280 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cv5q\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-kube-api-access-4cv5q\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243445 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243313 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-socket\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243445 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25418406-96a6-4eec-a5a4-98e672c5dcb5-istiod-ca-cert\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243572 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243445 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-token\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243572 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243461 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-certs\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243572 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-data\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243718 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243585 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-envoy\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243718 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243612 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-credential-socket\") pod \"25418406-96a6-4eec-a5a4-98e672c5dcb5\" (UID: \"25418406-96a6-4eec-a5a4-98e672c5dcb5\") " Sep 11 20:01:37.243851 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243707 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:37.243911 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243862 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-data" (OuterVolumeSpecName: "istio-data") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:37.243973 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243950 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:37.244036 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25418406-96a6-4eec-a5a4-98e672c5dcb5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:01:37.244036 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.243888 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:37.244204 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.244184 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.244261 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.244210 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-data\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.244261 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.244224 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-credential-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.244261 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.244240 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-workload-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.244261 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.244252 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25418406-96a6-4eec-a5a4-98e672c5dcb5-istiod-ca-cert\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.246045 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.246023 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:37.246167 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.246145 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-token" (OuterVolumeSpecName: "istio-token") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:37.246280 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.246256 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-kube-api-access-4cv5q" (OuterVolumeSpecName: "kube-api-access-4cv5q") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "kube-api-access-4cv5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:37.246346 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.246328 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25418406-96a6-4eec-a5a4-98e672c5dcb5" (UID: "25418406-96a6-4eec-a5a4-98e672c5dcb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:01:37.345532 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.345499 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-envoy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.345532 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.345529 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-podinfo\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.345532 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.345538 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cv5q\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-kube-api-access-4cv5q\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.345782 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.345547 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25418406-96a6-4eec-a5a4-98e672c5dcb5-istio-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:37.599460 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.599424 2579 generic.go:356] "Generic (PLEG): container finished" podID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerID="e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6" exitCode=0 Sep 11 20:01:37.599643 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.599490 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" Sep 11 20:01:37.599643 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.599487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" event={"ID":"25418406-96a6-4eec-a5a4-98e672c5dcb5","Type":"ContainerDied","Data":"e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6"} Sep 11 20:01:37.599643 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.599626 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv" event={"ID":"25418406-96a6-4eec-a5a4-98e672c5dcb5","Type":"ContainerDied","Data":"f67094af3551341b1dc2f5e33fb3b2417dcf8fd64a0592460e5fe17a59913f46"} Sep 11 20:01:37.599813 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.599657 2579 scope.go:117] "RemoveContainer" containerID="e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6" Sep 11 20:01:37.610488 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.610458 2579 scope.go:117] "RemoveContainer" containerID="be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb" Sep 11 20:01:37.619357 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.619328 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv"] Sep 11 20:01:37.622421 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.622375 2579 scope.go:117] "RemoveContainer" containerID="e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6" Sep 11 20:01:37.622793 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:01:37.622773 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6\": container with ID starting with e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6 not found: ID does not exist" containerID="e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6" Sep 11 20:01:37.622917 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.622803 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6"} err="failed to get container status \"e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6\": rpc error: code = NotFound desc = could not find container \"e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6\": container with ID starting with e5adfb39b4fcb1c2d668ec4ce0c1826dd62b666f5b6b3a8b9f52ee6679d784d6 not found: ID does not exist" Sep 11 20:01:37.622917 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.622836 2579 scope.go:117] "RemoveContainer" containerID="be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb" Sep 11 20:01:37.623166 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:01:37.623146 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb\": container with ID starting with be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb not found: ID does not exist" containerID="be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb" Sep 11 20:01:37.623217 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.623170 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb"} err="failed to get container status \"be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb\": rpc error: code = NotFound desc = could not find container \"be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb\": container with ID starting with be8db1ac34a3742910a21f7c767523fe4fcae8d64136996c0386feb714eef4fb not found: ID does not exist" Sep 11 20:01:37.626200 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.626175 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b1a074-openshift-default-74c5db6cfb-tmtqv"] Sep 11 20:01:37.879735 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:37.879640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:39.393939 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:39.393910 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" path="/var/lib/kubelet/pods/25418406-96a6-4eec-a5a4-98e672c5dcb5/volumes" Sep 11 20:01:40.184692 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:40.184663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:40.253876 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:40.253847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:40.415134 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:40.415097 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4"] Sep 11 20:01:40.415991 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:40.415957 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" containerID="cri-o://3f20fb590d5ddb6de2533f915e17c360404a2f2e67544d47b0c40682fbc38207" gracePeriod=30 Sep 11 20:01:40.442350 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:40.442261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:42.470427 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:42.470393 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.080259 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.080213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.130540 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.130509 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c"] Sep 11 20:01:43.130765 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.130726 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" podUID="6b3ace00-7762-46ca-b910-5ab54c59b690" containerName="payload-pre-processing" containerID="cri-o://174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939" gracePeriod=30 Sep 11 20:01:43.181238 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.181191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.280184 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.280104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.383008 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.382962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.414065 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.414041 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 20:01:43.519363 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.519322 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpkm7\" (UniqueName: \"kubernetes.io/projected/6b3ace00-7762-46ca-b910-5ab54c59b690-kube-api-access-gpkm7\") pod \"6b3ace00-7762-46ca-b910-5ab54c59b690\" (UID: \"6b3ace00-7762-46ca-b910-5ab54c59b690\") " Sep 11 20:01:43.519898 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.519397 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ace00-7762-46ca-b910-5ab54c59b690-plugins-config-volume\") pod \"6b3ace00-7762-46ca-b910-5ab54c59b690\" (UID: \"6b3ace00-7762-46ca-b910-5ab54c59b690\") " Sep 11 20:01:43.519898 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.519781 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3ace00-7762-46ca-b910-5ab54c59b690-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b3ace00-7762-46ca-b910-5ab54c59b690" (UID: "6b3ace00-7762-46ca-b910-5ab54c59b690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:01:43.520474 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.520445 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ace00-7762-46ca-b910-5ab54c59b690-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:43.521804 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.521785 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3ace00-7762-46ca-b910-5ab54c59b690-kube-api-access-gpkm7" (OuterVolumeSpecName: "kube-api-access-gpkm7") pod "6b3ace00-7762-46ca-b910-5ab54c59b690" (UID: "6b3ace00-7762-46ca-b910-5ab54c59b690"). InnerVolumeSpecName "kube-api-access-gpkm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:43.592668 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.592629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.622069 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.622028 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpkm7\" (UniqueName: \"kubernetes.io/projected/6b3ace00-7762-46ca-b910-5ab54c59b690-kube-api-access-gpkm7\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:43.640009 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.639917 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b3ace00-7762-46ca-b910-5ab54c59b690" containerID="174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939" exitCode=0 Sep 11 20:01:43.640009 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.639966 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" Sep 11 20:01:43.640009 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.640000 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" event={"ID":"6b3ace00-7762-46ca-b910-5ab54c59b690","Type":"ContainerDied","Data":"174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939"} Sep 11 20:01:43.640265 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.640051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c" event={"ID":"6b3ace00-7762-46ca-b910-5ab54c59b690","Type":"ContainerDied","Data":"dda5f6a84277d320269a0f9874957c2e497dae4a6eb1ffbefbd7a33a34ca99be"} Sep 11 20:01:43.640265 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.640076 2579 scope.go:117] "RemoveContainer" containerID="174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939" Sep 11 20:01:43.649582 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.649563 2579 scope.go:117] "RemoveContainer" containerID="174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939" Sep 11 20:01:43.650077 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:01:43.649974 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939\": container with ID starting with 174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939 not found: ID does not exist" containerID="174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939" Sep 11 20:01:43.650077 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.650013 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939"} err="failed to get container status \"174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939\": rpc error: code = NotFound desc = could not find container \"174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939\": container with ID starting with 174a5b6953d798af19ed6489ff7126a705fc23005d741c86d81fd6d2401bf939 not found: ID does not exist" Sep 11 20:01:43.665304 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.665268 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c"] Sep 11 20:01:43.665973 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.665951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:43.667806 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:43.667783 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e8cfcf-c557b9dcd-jxz8c"] Sep 11 20:01:44.294305 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:44.294276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:44.505673 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:44.505636 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 20:01:44.505900 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:44.505694 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 20:01:45.075092 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.075060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:45.394585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.394488 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3ace00-7762-46ca-b910-5ab54c59b690" path="/var/lib/kubelet/pods/6b3ace00-7762-46ca-b910-5ab54c59b690/volumes" Sep 11 20:01:45.655347 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.655256 2579 generic.go:356] "Generic (PLEG): container finished" podID="686e089b-fcff-45f8-8481-67d970a09e8e" containerID="3f20fb590d5ddb6de2533f915e17c360404a2f2e67544d47b0c40682fbc38207" exitCode=0 Sep 11 20:01:45.655347 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.655333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" event={"ID":"686e089b-fcff-45f8-8481-67d970a09e8e","Type":"ContainerDied","Data":"3f20fb590d5ddb6de2533f915e17c360404a2f2e67544d47b0c40682fbc38207"} Sep 11 20:01:45.655558 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.655397 2579 scope.go:117] "RemoveContainer" containerID="462851e3465c164a16247a7e9611da5e8a944eeeb4cc10406b481c2cf63c2385" Sep 11 20:01:45.688262 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.688239 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 20:01:45.746363 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746334 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686e089b-fcff-45f8-8481-67d970a09e8e-istiod-ca-cert\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746538 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746404 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-certs\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746538 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746437 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-credential-socket\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746538 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746476 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686e089b-fcff-45f8-8481-67d970a09e8e-istio-podinfo\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746538 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746508 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-istio-token\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746538 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746529 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-envoy\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746862 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746562 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-socket\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746862 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746614 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-data\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746862 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746668 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjjp5\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-kube-api-access-jjjp5\") pod \"686e089b-fcff-45f8-8481-67d970a09e8e\" (UID: \"686e089b-fcff-45f8-8481-67d970a09e8e\") " Sep 11 20:01:45.746862 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746774 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:45.746862 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746793 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:45.746862 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.746803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686e089b-fcff-45f8-8481-67d970a09e8e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:01:45.747251 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.747013 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:45.747251 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.747239 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.747370 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.747263 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-credential-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.747370 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.747277 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-workload-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.747370 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.747290 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686e089b-fcff-45f8-8481-67d970a09e8e-istiod-ca-cert\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.747565 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.747545 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-data" (OuterVolumeSpecName: "istio-data") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:45.748917 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.748888 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:45.749560 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.749539 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/686e089b-fcff-45f8-8481-67d970a09e8e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:01:45.749616 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.749559 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-istio-token" (OuterVolumeSpecName: "istio-token") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:45.749709 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.749688 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-kube-api-access-jjjp5" (OuterVolumeSpecName: "kube-api-access-jjjp5") pod "686e089b-fcff-45f8-8481-67d970a09e8e" (UID: "686e089b-fcff-45f8-8481-67d970a09e8e"). InnerVolumeSpecName "kube-api-access-jjjp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:45.848724 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.848685 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-istio-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.848724 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.848716 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-envoy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.848724 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.848725 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686e089b-fcff-45f8-8481-67d970a09e8e-istio-data\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.848724 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.848733 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjp5\" (UniqueName: \"kubernetes.io/projected/686e089b-fcff-45f8-8481-67d970a09e8e-kube-api-access-jjjp5\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.849036 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.848742 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686e089b-fcff-45f8-8481-67d970a09e8e-istio-podinfo\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:45.897440 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:45.897414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:46.662221 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:46.662188 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" Sep 11 20:01:46.662663 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:46.662231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4" event={"ID":"686e089b-fcff-45f8-8481-67d970a09e8e","Type":"ContainerDied","Data":"f10e5c972883ee142b281352972cd628abd16c46450299a66849dbe6ff703164"} Sep 11 20:01:46.662663 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:46.662304 2579 scope.go:117] "RemoveContainer" containerID="3f20fb590d5ddb6de2533f915e17c360404a2f2e67544d47b0c40682fbc38207" Sep 11 20:01:46.686391 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:46.686350 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4"] Sep 11 20:01:46.691028 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:46.691004 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a8ee95-openshift-default-79bcb67594-qw5h4"] Sep 11 20:01:47.394235 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:47.394201 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" path="/var/lib/kubelet/pods/686e089b-fcff-45f8-8481-67d970a09e8e/volumes" Sep 11 20:01:49.405254 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:49.405222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:49.477876 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:49.477818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:49.634136 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:49.634102 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn"] Sep 11 20:01:49.634376 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:49.634348 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" containerID="cri-o://11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2" gracePeriod=30 Sep 11 20:01:49.657989 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:49.657882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:50.062351 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:50.062315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:50.160842 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:50.160764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:50.964802 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:50.964770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:51.661686 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:51.661632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:52.350560 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:52.350526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:53.115739 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:53.115707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:54.897320 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:54.897261 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 20:01:55.046819 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.046776 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-podinfo\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047032 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.046915 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-socket\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047032 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.046954 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-data\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047032 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.046989 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-credential-socket\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047204 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047046 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgk6d\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-kube-api-access-hgk6d\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047204 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047087 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-envoy\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047204 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047122 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istiod-ca-cert\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047204 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047187 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-certs\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047420 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047248 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-token\") pod \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\" (UID: \"5f9940a5-44e6-479c-a5bb-c89c837ea9d2\") " Sep 11 20:01:55.047420 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047247 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:55.047420 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047261 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:55.047420 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:55.047614 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047475 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:55.047614 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047543 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:01:55.047721 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047649 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.047721 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047663 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-workload-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.047721 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047671 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-data\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.047721 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047679 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-credential-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.047721 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.047688 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istiod-ca-cert\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.049509 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.049476 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-kube-api-access-hgk6d" (OuterVolumeSpecName: "kube-api-access-hgk6d") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "kube-api-access-hgk6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:55.049639 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.049520 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:01:55.049639 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.049559 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:01:55.049639 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.049590 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f9940a5-44e6-479c-a5bb-c89c837ea9d2" (UID: "5f9940a5-44e6-479c-a5bb-c89c837ea9d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:55.148500 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.148407 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.148500 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.148440 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-podinfo\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.148500 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.148449 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgk6d\" (UniqueName: \"kubernetes.io/projected/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-kube-api-access-hgk6d\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.148500 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.148458 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f9940a5-44e6-479c-a5bb-c89c837ea9d2-istio-envoy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:01:55.715703 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.715599 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerID="11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2" exitCode=0 Sep 11 20:01:55.715703 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.715647 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" Sep 11 20:01:55.715703 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.715692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" event={"ID":"5f9940a5-44e6-479c-a5bb-c89c837ea9d2","Type":"ContainerDied","Data":"11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2"} Sep 11 20:01:55.715956 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.715748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn" event={"ID":"5f9940a5-44e6-479c-a5bb-c89c837ea9d2","Type":"ContainerDied","Data":"c2482a8a3613236890c830d1153c56f645d62ceb0a12ee5a0007b0415a46936f"} Sep 11 20:01:55.715956 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.715779 2579 scope.go:117] "RemoveContainer" containerID="11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2" Sep 11 20:01:55.725346 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.725324 2579 scope.go:117] "RemoveContainer" containerID="276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5" Sep 11 20:01:55.734677 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.734658 2579 scope.go:117] "RemoveContainer" containerID="11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2" Sep 11 20:01:55.735017 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:01:55.734997 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2\": container with ID starting with 11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2 not found: ID does not exist" containerID="11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2" Sep 11 20:01:55.735102 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.735027 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2"} err="failed to get container status \"11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2\": rpc error: code = NotFound desc = could not find container \"11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2\": container with ID starting with 11dfb2dfdb2ed12c9d8bf7c1db17b49c8dfc505b2365c6747fad11543f0251d2 not found: ID does not exist" Sep 11 20:01:55.735102 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.735048 2579 scope.go:117] "RemoveContainer" containerID="276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5" Sep 11 20:01:55.735317 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:01:55.735295 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5\": container with ID starting with 276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5 not found: ID does not exist" containerID="276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5" Sep 11 20:01:55.735420 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.735321 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5"} err="failed to get container status \"276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5\": rpc error: code = NotFound desc = could not find container \"276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5\": container with ID starting with 276a3a00f07b5ab32f2bdc5fb151cb5ab0660c671b0c5b086890c9569df44af5 not found: ID does not exist" Sep 11 20:01:55.757880 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.757841 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn"] Sep 11 20:01:55.760336 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:55.760307 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e8cfcf-openshift-default-6bdc48cf5b-kj2nn"] Sep 11 20:01:57.394777 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:57.394741 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" path="/var/lib/kubelet/pods/5f9940a5-44e6-479c-a5bb-c89c837ea9d2/volumes" Sep 11 20:01:59.227578 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:59.227542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:01:59.289933 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:01:59.289897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:00.114222 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.114192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:00.491180 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.491074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:00.493145 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.493120 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz"] Sep 11 20:02:00.496724 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.496695 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f7fd24-29819280-wbqpz"] Sep 11 20:02:00.510542 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.510493 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n"] Sep 11 20:02:00.510806 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.510776 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" podUID="234dd7fd-2d83-47d1-8ffe-f028db4d536b" containerName="payload-processing" containerID="cri-o://fa55afd203a2eb04c1cee5c0cc81157fd5b842f201f243871c129512136aca1b" gracePeriod=30 Sep 11 20:02:00.590816 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.590783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:00.675611 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.675577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:00.733300 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.733271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:00.775359 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.775327 2579 generic.go:356] "Generic (PLEG): container finished" podID="234dd7fd-2d83-47d1-8ffe-f028db4d536b" containerID="fa55afd203a2eb04c1cee5c0cc81157fd5b842f201f243871c129512136aca1b" exitCode=0 Sep 11 20:02:00.775562 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.775380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" event={"ID":"234dd7fd-2d83-47d1-8ffe-f028db4d536b","Type":"ContainerDied","Data":"fa55afd203a2eb04c1cee5c0cc81157fd5b842f201f243871c129512136aca1b"} Sep 11 20:02:00.806062 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.806036 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 20:02:00.906510 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.906469 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234dd7fd-2d83-47d1-8ffe-f028db4d536b-plugins-config-volume\") pod \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\" (UID: \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\") " Sep 11 20:02:00.906723 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.906634 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcgsv\" (UniqueName: \"kubernetes.io/projected/234dd7fd-2d83-47d1-8ffe-f028db4d536b-kube-api-access-hcgsv\") pod \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\" (UID: \"234dd7fd-2d83-47d1-8ffe-f028db4d536b\") " Sep 11 20:02:00.906935 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.906907 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234dd7fd-2d83-47d1-8ffe-f028db4d536b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "234dd7fd-2d83-47d1-8ffe-f028db4d536b" (UID: "234dd7fd-2d83-47d1-8ffe-f028db4d536b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:02:00.907085 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.907065 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234dd7fd-2d83-47d1-8ffe-f028db4d536b-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:00.909527 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:00.909484 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234dd7fd-2d83-47d1-8ffe-f028db4d536b-kube-api-access-hcgsv" (OuterVolumeSpecName: "kube-api-access-hcgsv") pod "234dd7fd-2d83-47d1-8ffe-f028db4d536b" (UID: "234dd7fd-2d83-47d1-8ffe-f028db4d536b"). InnerVolumeSpecName "kube-api-access-hcgsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:02:01.008411 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.008366 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgsv\" (UniqueName: \"kubernetes.io/projected/234dd7fd-2d83-47d1-8ffe-f028db4d536b-kube-api-access-hcgsv\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:01.017502 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.017471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:01.118258 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.118229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:01.393907 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.393802 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df339cb7-d4d2-4d55-b124-d1d0917a4a97" path="/var/lib/kubelet/pods/df339cb7-d4d2-4d55-b124-d1d0917a4a97/volumes" Sep 11 20:02:01.782743 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.782651 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" Sep 11 20:02:01.782743 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.782681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n" event={"ID":"234dd7fd-2d83-47d1-8ffe-f028db4d536b","Type":"ContainerDied","Data":"b942aa46081f5c93d062ad2234a7a93228c3c34c0f5ee5f2f72232798ed96239"} Sep 11 20:02:01.782743 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.782734 2579 scope.go:117] "RemoveContainer" containerID="fa55afd203a2eb04c1cee5c0cc81157fd5b842f201f243871c129512136aca1b" Sep 11 20:02:01.801789 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.801758 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n"] Sep 11 20:02:01.806813 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.806784 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f7fd24-7cf8bb44d5-ncz8n"] Sep 11 20:02:01.906147 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:01.906108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:02.691496 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:02.691443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:03.394323 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:03.394278 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234dd7fd-2d83-47d1-8ffe-f028db4d536b" path="/var/lib/kubelet/pods/234dd7fd-2d83-47d1-8ffe-f028db4d536b/volumes" Sep 11 20:02:05.940981 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:05.940949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:06.863834 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:06.863785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:06.964550 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:06.964520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:07.115347 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:07.115261 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4"] Sep 11 20:02:07.115513 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:07.115488 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" containerID="cri-o://3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a" gracePeriod=30 Sep 11 20:02:07.143776 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:07.143747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:08.925593 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:08.925555 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 20:02:08.925982 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:08.925616 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 20:02:12.379600 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.379535 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 20:02:12.424720 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.424677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8mq9\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-kube-api-access-t8mq9\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.424720 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.424723 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-envoy\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.425000 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.424768 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-credential-socket\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.425000 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.424843 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-socket\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.425000 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.424908 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-certs\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.425000 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.424942 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-data\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.425212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425013 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c206e47f-3c55-4953-ad34-34f1e242579e-istio-podinfo\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.425212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425060 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c206e47f-3c55-4953-ad34-34f1e242579e-istiod-ca-cert\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.425212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425095 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-istio-token\") pod \"c206e47f-3c55-4953-ad34-34f1e242579e\" (UID: \"c206e47f-3c55-4953-ad34-34f1e242579e\") " Sep 11 20:02:12.425354 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425262 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:02:12.425448 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425403 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:02:12.425727 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425554 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.425727 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425618 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:02:12.425727 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425667 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:02:12.425727 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425701 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-data\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.426210 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.425732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c206e47f-3c55-4953-ad34-34f1e242579e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:02:12.427587 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.427560 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-kube-api-access-t8mq9" (OuterVolumeSpecName: "kube-api-access-t8mq9") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "kube-api-access-t8mq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:02:12.427814 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.427781 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:02:12.428070 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.428047 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:02:12.428070 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.428050 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c206e47f-3c55-4953-ad34-34f1e242579e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c206e47f-3c55-4953-ad34-34f1e242579e" (UID: "c206e47f-3c55-4953-ad34-34f1e242579e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:02:12.526355 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.526318 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-workload-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.526355 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.526344 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c206e47f-3c55-4953-ad34-34f1e242579e-istio-podinfo\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.526355 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.526354 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c206e47f-3c55-4953-ad34-34f1e242579e-istiod-ca-cert\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.526355 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.526367 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-istio-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.526663 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.526377 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8mq9\" (UniqueName: \"kubernetes.io/projected/c206e47f-3c55-4953-ad34-34f1e242579e-kube-api-access-t8mq9\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.526663 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.526385 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-istio-envoy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.526663 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.526393 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c206e47f-3c55-4953-ad34-34f1e242579e-credential-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:02:12.845482 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.845448 2579 generic.go:356] "Generic (PLEG): container finished" podID="c206e47f-3c55-4953-ad34-34f1e242579e" containerID="3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a" exitCode=0 Sep 11 20:02:12.845680 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.845498 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" Sep 11 20:02:12.845680 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.845543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" event={"ID":"c206e47f-3c55-4953-ad34-34f1e242579e","Type":"ContainerDied","Data":"3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a"} Sep 11 20:02:12.845680 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.845596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4" event={"ID":"c206e47f-3c55-4953-ad34-34f1e242579e","Type":"ContainerDied","Data":"b81ea5318883df5f95f97a9188f8a4c37530003c32e5afe30faa5c523fcbd02f"} Sep 11 20:02:12.845680 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.845619 2579 scope.go:117] "RemoveContainer" containerID="3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a" Sep 11 20:02:12.855613 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.855594 2579 scope.go:117] "RemoveContainer" containerID="ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3" Sep 11 20:02:12.865227 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.865201 2579 scope.go:117] "RemoveContainer" containerID="3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a" Sep 11 20:02:12.865614 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:02:12.865588 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a\": container with ID starting with 3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a not found: ID does not exist" containerID="3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a" Sep 11 20:02:12.865709 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.865621 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a"} err="failed to get container status \"3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a\": rpc error: code = NotFound desc = could not find container \"3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a\": container with ID starting with 3e3100abc2fa1a7345fcd87dc9b8149b2fd4dc4862f2fa09f5cecad8d8ec2b5a not found: ID does not exist" Sep 11 20:02:12.865709 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.865644 2579 scope.go:117] "RemoveContainer" containerID="ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3" Sep 11 20:02:12.866135 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:02:12.866105 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3\": container with ID starting with ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3 not found: ID does not exist" containerID="ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3" Sep 11 20:02:12.866277 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.866142 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3"} err="failed to get container status \"ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3\": rpc error: code = NotFound desc = could not find container \"ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3\": container with ID starting with ece38c27b169c09ac3263dccd7c2c67be38f8da8f7754a2eed281e2e8eecd0a3 not found: ID does not exist" Sep 11 20:02:12.869181 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.869160 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4"] Sep 11 20:02:12.873380 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:12.873355 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f7fd24-openshift-default-b7d7fcb6b-rcfv4"] Sep 11 20:02:13.393613 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:13.393572 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" path="/var/lib/kubelet/pods/c206e47f-3c55-4953-ad34-34f1e242579e/volumes" Sep 11 20:02:15.289231 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:15.289203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:16.189124 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:16.189092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:23.975856 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:23.975743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:02:46.590495 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:02:46.590463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:28.895647 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:03:28.895614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:31.487726 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:03:31.487688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:32.483913 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:03:32.483876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:50.586197 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:03:50.586117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:03:56.958048 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:03:56.958003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:08.658602 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:08.658540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:12.081534 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:12.081499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:27.514276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:27.514249 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 20:04:27.516191 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:27.516169 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 20:04:46.116138 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:46.116057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:47.260589 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:47.260554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:47.667107 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:47.667065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:47.973271 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:47.973176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:48.214348 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:48.214308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:50.860285 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:50.860254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:51.176948 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:51.176865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:51.283316 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:51.283283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:51.777586 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:51.777555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:52.705860 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:52.705805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:52.787923 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:52.787885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:53.069404 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.069368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:53.343857 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.343739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:53.454616 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.454572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:53.455728 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.455706 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g"] Sep 11 20:04:53.456167 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456153 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234dd7fd-2d83-47d1-8ffe-f028db4d536b" containerName="payload-processing" Sep 11 20:04:53.456212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456168 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="234dd7fd-2d83-47d1-8ffe-f028db4d536b" containerName="payload-processing" Sep 11 20:04:53.456212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456181 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3ace00-7762-46ca-b910-5ab54c59b690" containerName="payload-pre-processing" Sep 11 20:04:53.456212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456186 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3ace00-7762-46ca-b910-5ab54c59b690" containerName="payload-pre-processing" Sep 11 20:04:53.456212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456195 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" Sep 11 20:04:53.456212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456200 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" Sep 11 20:04:53.456212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456210 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f471704e-af44-400e-bb99-cdff6e6fbea7" containerName="payload-processing" Sep 11 20:04:53.456212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456215 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f471704e-af44-400e-bb99-cdff6e6fbea7" containerName="payload-processing" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456222 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456228 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456236 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456241 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456262 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146408ba-5617-47ee-adba-3baf420d38d1" containerName="payload-pre-processing" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456269 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="146408ba-5617-47ee-adba-3baf420d38d1" containerName="payload-pre-processing" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456277 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456282 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456295 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92496486-fd20-4fc7-bd43-03609f74556b" containerName="payload-processing" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456300 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="92496486-fd20-4fc7-bd43-03609f74556b" containerName="payload-processing" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456306 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456311 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456318 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371ac68d-4503-4374-b598-033d7687e0d1" containerName="payload-processing" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456323 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="371ac68d-4503-4374-b598-033d7687e0d1" containerName="payload-processing" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456330 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456335 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456340 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456345 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456403 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f471704e-af44-400e-bb99-cdff6e6fbea7" containerName="payload-processing" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456412 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456418 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456424 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" Sep 11 20:04:53.456424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456430 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" Sep 11 20:04:53.457190 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456438 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3ace00-7762-46ca-b910-5ab54c59b690" containerName="payload-pre-processing" Sep 11 20:04:53.457190 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456444 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="146408ba-5617-47ee-adba-3baf420d38d1" containerName="payload-pre-processing" Sep 11 20:04:53.457190 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456451 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" Sep 11 20:04:53.457190 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456457 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="92496486-fd20-4fc7-bd43-03609f74556b" containerName="payload-processing" Sep 11 20:04:53.457190 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456463 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="25418406-96a6-4eec-a5a4-98e672c5dcb5" containerName="istio-proxy" Sep 11 20:04:53.457190 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456469 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="371ac68d-4503-4374-b598-033d7687e0d1" containerName="payload-processing" Sep 11 20:04:53.457190 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.456473 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="234dd7fd-2d83-47d1-8ffe-f028db4d536b" containerName="payload-processing" Sep 11 20:04:53.459664 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.459647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:53.464274 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.464253 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-96e341\"" Sep 11 20:04:53.471672 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.471647 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g"] Sep 11 20:04:53.484729 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.484691 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt"] Sep 11 20:04:53.485169 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.485154 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" Sep 11 20:04:53.485169 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.485169 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c206e47f-3c55-4953-ad34-34f1e242579e" containerName="istio-proxy" Sep 11 20:04:53.485308 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.485243 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="686e089b-fcff-45f8-8481-67d970a09e8e" containerName="istio-proxy" Sep 11 20:04:53.485308 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.485251 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f9940a5-44e6-479c-a5bb-c89c837ea9d2" containerName="istio-proxy" Sep 11 20:04:53.488293 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.488275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:53.491099 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.491080 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-96e341-dockercfg-jhmsw\"" Sep 11 20:04:53.501971 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.501948 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt"] Sep 11 20:04:53.568885 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.568847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5792\" (UniqueName: \"kubernetes.io/projected/7d262392-3996-48ab-9585-3355d9b97e17-kube-api-access-t5792\") pod \"payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt\" (UID: \"7d262392-3996-48ab-9585-3355d9b97e17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:53.569080 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.568926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262392-3996-48ab-9585-3355d9b97e17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt\" (UID: \"7d262392-3996-48ab-9585-3355d9b97e17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:53.569140 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.569095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g\" (UID: \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:53.569195 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.569171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7rt\" (UniqueName: \"kubernetes.io/projected/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-kube-api-access-tv7rt\") pod \"payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g\" (UID: \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:53.669940 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.669852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g\" (UID: \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:53.669940 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.669908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7rt\" (UniqueName: \"kubernetes.io/projected/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-kube-api-access-tv7rt\") pod \"payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g\" (UID: \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:53.670132 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.669963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5792\" (UniqueName: \"kubernetes.io/projected/7d262392-3996-48ab-9585-3355d9b97e17-kube-api-access-t5792\") pod \"payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt\" (UID: \"7d262392-3996-48ab-9585-3355d9b97e17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:53.670132 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.669982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262392-3996-48ab-9585-3355d9b97e17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt\" (UID: \"7d262392-3996-48ab-9585-3355d9b97e17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:53.671849 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.670913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g\" (UID: \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:53.671849 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.671237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262392-3996-48ab-9585-3355d9b97e17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt\" (UID: \"7d262392-3996-48ab-9585-3355d9b97e17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:53.681034 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.680993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5792\" (UniqueName: \"kubernetes.io/projected/7d262392-3996-48ab-9585-3355d9b97e17-kube-api-access-t5792\") pod \"payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt\" (UID: \"7d262392-3996-48ab-9585-3355d9b97e17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:53.681034 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.681019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7rt\" (UniqueName: \"kubernetes.io/projected/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-kube-api-access-tv7rt\") pod \"payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g\" (UID: \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:53.772013 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.771975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:53.796321 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.796289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:53.799283 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:53.799255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:54.049388 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.049361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g"] Sep 11 20:04:54.051399 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:04:54.051369 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbd04b7_84b4_452b_b6e3_b05f12fd890e.slice/crio-278f827d5060a924eea642d5f8e01ff2413355e3fc57e50214e2aff4633e3a30 WatchSource:0}: Error finding container 278f827d5060a924eea642d5f8e01ff2413355e3fc57e50214e2aff4633e3a30: Status 404 returned error can't find the container with id 278f827d5060a924eea642d5f8e01ff2413355e3fc57e50214e2aff4633e3a30 Sep 11 20:04:54.094542 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.094515 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt"] Sep 11 20:04:54.095960 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:04:54.095928 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d262392_3996_48ab_9585_3355d9b97e17.slice/crio-01921e6355287cb5e0e89ae0284825e75b3e9843a5d883cdfcc60567aa85f715 WatchSource:0}: Error finding container 01921e6355287cb5e0e89ae0284825e75b3e9843a5d883cdfcc60567aa85f715: Status 404 returned error can't find the container with id 01921e6355287cb5e0e89ae0284825e75b3e9843a5d883cdfcc60567aa85f715 Sep 11 20:04:54.559536 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.559502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:04:54.727530 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.727492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" event={"ID":"5bbd04b7-84b4-452b-b6e3-b05f12fd890e","Type":"ContainerStarted","Data":"90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df"} Sep 11 20:04:54.727722 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.727559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:04:54.727722 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.727586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" event={"ID":"5bbd04b7-84b4-452b-b6e3-b05f12fd890e","Type":"ContainerStarted","Data":"278f827d5060a924eea642d5f8e01ff2413355e3fc57e50214e2aff4633e3a30"} Sep 11 20:04:54.729299 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.729274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" event={"ID":"7d262392-3996-48ab-9585-3355d9b97e17","Type":"ContainerStarted","Data":"26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414"} Sep 11 20:04:54.729411 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.729315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" event={"ID":"7d262392-3996-48ab-9585-3355d9b97e17","Type":"ContainerStarted","Data":"01921e6355287cb5e0e89ae0284825e75b3e9843a5d883cdfcc60567aa85f715"} Sep 11 20:04:54.729450 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.729415 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:04:54.752617 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.752569 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" podStartSLOduration=1.752554825 podStartE2EDuration="1.752554825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:04:54.750206128 +0000 UTC m=+2127.883518717" watchObservedRunningTime="2026-09-11 20:04:54.752554825 +0000 UTC m=+2127.885867414" Sep 11 20:04:54.770859 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:04:54.770787 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" podStartSLOduration=1.770767759 podStartE2EDuration="1.770767759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:04:54.769219294 +0000 UTC m=+2127.902531894" watchObservedRunningTime="2026-09-11 20:04:54.770767759 +0000 UTC m=+2127.904080352" Sep 11 20:05:05.737528 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:05.737491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:05:05.738054 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:05.737581 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:05:14.693292 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:14.693254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:15.450163 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:15.450124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:16.067465 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:16.067431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:16.170025 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:16.169993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:16.741175 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:16.741139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:16.841967 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:16.841936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:17.137298 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:17.137266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:17.235277 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:17.235239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:19.599686 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:19.599598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:20.310909 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:20.310876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:20.453803 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:20.453769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:20.571085 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:20.570996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:21.454136 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:21.454107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:22.307063 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:22.307028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:23.042461 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:23.042425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:23.140054 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:23.140021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:23.495603 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:23.495517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:23.600044 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:23.600007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:23.672074 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:23.672038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:27.678602 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:27.678572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:32.439090 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:32.439051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:32.538978 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:32.538943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:43.859751 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:43.859690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:45.150340 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:45.150304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:45.270869 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:45.270840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:46.012468 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:46.012435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:46.702417 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:46.702385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:47.406378 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:47.406345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:48.182401 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:48.182357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:50.210988 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:50.210953 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2ht6_9d9a0f88-e534-415a-bfea-a3e605edc458/payload-processing/0.log" Sep 11 20:05:55.833716 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:55.833680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:05:57.472850 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:05:57.472781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:20.857614 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:20.857582 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt_7d262392-3996-48ab-9585-3355d9b97e17/payload-processing/0.log" Sep 11 20:06:20.983989 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:20.983960 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2ht6_9d9a0f88-e534-415a-bfea-a3e605edc458/payload-processing/0.log" Sep 11 20:06:21.377490 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.377460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:21.752652 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.752567 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn"] Sep 11 20:06:21.756442 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.756423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.760085 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.760063 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-2f8fd9-openshift-default-dockercfg-qpzf5\"" Sep 11 20:06:21.793291 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.793247 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn"] Sep 11 20:06:21.820602 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.820573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:21.894378 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-credential-socket\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.894760 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-certs\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.894760 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-token\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.894760 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-podinfo\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.894760 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-envoy\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.894760 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-socket\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.894760 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-data\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.894760 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315cea54-3be9-4612-9a3a-75becf2a9d11-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.894760 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.894710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r69f\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-kube-api-access-9r69f\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.995889 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.995855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-certs\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996078 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.995897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-token\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996078 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.995928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-podinfo\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996078 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.995983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-envoy\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996078 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-socket\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996078 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-data\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996472 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315cea54-3be9-4612-9a3a-75becf2a9d11-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996472 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r69f\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-kube-api-access-9r69f\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996472 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-credential-socket\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996472 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996375 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-certs\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996696 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-data\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996756 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-socket\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.996856 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.996815 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-credential-socket\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.997212 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.997191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315cea54-3be9-4612-9a3a-75becf2a9d11-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.998595 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.998576 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-envoy\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:21.998764 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:21.998741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-podinfo\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:22.009489 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:22.009412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r69f\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-kube-api-access-9r69f\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:22.009781 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:22.009765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-token\") pod \"e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:22.042331 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:22.042294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:22.067172 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:22.067134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:22.354171 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:22.354146 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn"] Sep 11 20:06:22.355528 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:06:22.355506 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315cea54_3be9_4612_9a3a_75becf2a9d11.slice/crio-b8544f8bfb8eb33bd7eaa93357799ff5aa426e62d129e0e71b8dc645757644ed WatchSource:0}: Error finding container b8544f8bfb8eb33bd7eaa93357799ff5aa426e62d129e0e71b8dc645757644ed: Status 404 returned error can't find the container with id b8544f8bfb8eb33bd7eaa93357799ff5aa426e62d129e0e71b8dc645757644ed Sep 11 20:06:22.357814 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:22.357782 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:06:22.357946 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:22.357865 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:06:22.357946 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:22.357911 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:06:23.251734 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:23.251703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" event={"ID":"315cea54-3be9-4612-9a3a-75becf2a9d11","Type":"ContainerStarted","Data":"1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2"} Sep 11 20:06:23.252206 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:23.251758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" event={"ID":"315cea54-3be9-4612-9a3a-75becf2a9d11","Type":"ContainerStarted","Data":"b8544f8bfb8eb33bd7eaa93357799ff5aa426e62d129e0e71b8dc645757644ed"} Sep 11 20:06:23.275074 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:23.275014 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" podStartSLOduration=2.274998323 podStartE2EDuration="2.274998323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:23.274248014 +0000 UTC m=+2216.407560598" watchObservedRunningTime="2026-09-11 20:06:23.274998323 +0000 UTC m=+2216.408310911" Sep 11 20:06:24.067946 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:24.067900 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:24.073338 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:24.073309 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:24.257424 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:24.257390 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:24.258469 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:24.258451 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:24.513866 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:24.513753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:25.137804 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:25.137764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:25.234525 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:25.234486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:25.408398 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:25.408305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:26.292137 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:26.292100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:26.392462 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:26.392430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:27.183282 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:27.183251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:27.756303 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:27.756273 2579 scope.go:117] "RemoveContainer" containerID="4f031ec3f0440b63736998689b468bc588328e35a22f2a1ff731bca72c2875fe" Sep 11 20:06:27.765160 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:27.765136 2579 scope.go:117] "RemoveContainer" containerID="c8880a701c54358e1f9e7b4ccc1b32053f76a5c39c49b712ea60b27206f0a0ed" Sep 11 20:06:27.804348 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:27.804321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:28.479499 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:28.479464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:28.586881 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:28.586810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:28.649198 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:28.649164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:29.092230 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.092180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:29.220805 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.220774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:29.271466 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.271433 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z"] Sep 11 20:06:29.276471 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.276443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:29.279082 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.279062 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2f8fd9\"" Sep 11 20:06:29.279213 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.279091 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2f8fd9-dockercfg-xns7m\"" Sep 11 20:06:29.284690 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.284665 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z"] Sep 11 20:06:29.373611 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.373496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9ed380-e785-4bb7-9220-80950a405f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z\" (UID: \"6d9ed380-e785-4bb7-9220-80950a405f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:29.373611 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.373597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmzn7\" (UniqueName: \"kubernetes.io/projected/6d9ed380-e785-4bb7-9220-80950a405f70-kube-api-access-pmzn7\") pod \"payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z\" (UID: \"6d9ed380-e785-4bb7-9220-80950a405f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:29.474320 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.474285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9ed380-e785-4bb7-9220-80950a405f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z\" (UID: \"6d9ed380-e785-4bb7-9220-80950a405f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:29.474507 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.474338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmzn7\" (UniqueName: \"kubernetes.io/projected/6d9ed380-e785-4bb7-9220-80950a405f70-kube-api-access-pmzn7\") pod \"payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z\" (UID: \"6d9ed380-e785-4bb7-9220-80950a405f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:29.475034 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.475015 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9ed380-e785-4bb7-9220-80950a405f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z\" (UID: \"6d9ed380-e785-4bb7-9220-80950a405f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:29.484045 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.484023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmzn7\" (UniqueName: \"kubernetes.io/projected/6d9ed380-e785-4bb7-9220-80950a405f70-kube-api-access-pmzn7\") pod \"payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z\" (UID: \"6d9ed380-e785-4bb7-9220-80950a405f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:29.590069 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.590032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:29.863863 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:29.863837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z"] Sep 11 20:06:29.866046 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:06:29.866020 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9ed380_e785_4bb7_9220_80950a405f70.slice/crio-3b68f4ea145ea8b9b8387a64bc8f3086ce867769d3d0144243f487fd2fe1cc88 WatchSource:0}: Error finding container 3b68f4ea145ea8b9b8387a64bc8f3086ce867769d3d0144243f487fd2fe1cc88: Status 404 returned error can't find the container with id 3b68f4ea145ea8b9b8387a64bc8f3086ce867769d3d0144243f487fd2fe1cc88 Sep 11 20:06:30.296992 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:30.296956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" event={"ID":"6d9ed380-e785-4bb7-9220-80950a405f70","Type":"ContainerStarted","Data":"e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152"} Sep 11 20:06:30.297221 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:30.297036 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:30.297221 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:30.297060 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" event={"ID":"6d9ed380-e785-4bb7-9220-80950a405f70","Type":"ContainerStarted","Data":"3b68f4ea145ea8b9b8387a64bc8f3086ce867769d3d0144243f487fd2fe1cc88"} Sep 11 20:06:30.314562 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:30.314498 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" podStartSLOduration=1.31447971 podStartE2EDuration="1.31447971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:30.314124195 +0000 UTC m=+2223.447436794" watchObservedRunningTime="2026-09-11 20:06:30.31447971 +0000 UTC m=+2223.447792366" Sep 11 20:06:41.305022 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:41.304990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:06:44.376287 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:44.376198 2579 generic.go:356] "Generic (PLEG): container finished" podID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerID="1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2" exitCode=0 Sep 11 20:06:44.376287 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:44.376274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" event={"ID":"315cea54-3be9-4612-9a3a-75becf2a9d11","Type":"ContainerDied","Data":"1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2"} Sep 11 20:06:44.376732 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:44.376717 2579 scope.go:117] "RemoveContainer" containerID="1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2" Sep 11 20:06:44.377861 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:44.377834 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:06:44.377918 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:44.377901 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:06:44.377956 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:44.377932 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 20:06:45.394243 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:45.394205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" event={"ID":"315cea54-3be9-4612-9a3a-75becf2a9d11","Type":"ContainerStarted","Data":"18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7"} Sep 11 20:06:46.068175 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:46.068137 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:46.068403 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:46.068190 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 11 20:06:46.068403 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:46.068249 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 11 20:06:47.068231 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:47.068182 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 11 20:06:47.068756 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:47.068270 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 11 20:06:48.068106 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:48.068055 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 11 20:06:48.068345 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:48.068164 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 11 20:06:49.068383 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:49.068341 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 11 20:06:49.069034 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:49.068994 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 11 20:06:50.068046 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:50.068003 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 11 20:06:50.068355 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:50.068086 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 11 20:06:50.499403 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:50.499275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:51.072486 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:51.072448 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:51.072883 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:51.072861 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:51.073636 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:51.073616 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:06:51.287429 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:51.287392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:51.901271 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:51.901199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:52.687863 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:52.687788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:53.316243 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:53.316209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:53.992860 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:53.992805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:55.210030 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.209978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:55.850398 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.850364 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn"] Sep 11 20:06:55.853048 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.853032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" Sep 11 20:06:55.855833 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.855801 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 20:06:55.855922 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.855815 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 20:06:55.862008 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.861966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6890cb08-fe65-45db-bb41-1fee110f9a18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn\" (UID: \"6890cb08-fe65-45db-bb41-1fee110f9a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" Sep 11 20:06:55.863510 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.863476 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn"] Sep 11 20:06:55.963351 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.963316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6890cb08-fe65-45db-bb41-1fee110f9a18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn\" (UID: \"6890cb08-fe65-45db-bb41-1fee110f9a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" Sep 11 20:06:55.964019 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:55.964000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6890cb08-fe65-45db-bb41-1fee110f9a18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn\" (UID: \"6890cb08-fe65-45db-bb41-1fee110f9a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" Sep 11 20:06:56.165169 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:56.165080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" Sep 11 20:06:56.438795 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:56.438723 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn"] Sep 11 20:06:56.442444 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:06:56.442414 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6890cb08_fe65_45db_bb41_1fee110f9a18.slice/crio-18fce44600ac90af17cd8650e0f8532d52005b2f96c058e3de6759eda42a3c41 WatchSource:0}: Error finding container 18fce44600ac90af17cd8650e0f8532d52005b2f96c058e3de6759eda42a3c41: Status 404 returned error can't find the container with id 18fce44600ac90af17cd8650e0f8532d52005b2f96c058e3de6759eda42a3c41 Sep 11 20:06:56.653675 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:56.653633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:57.388130 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:57.388077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:57.457412 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:57.457301 2579 generic.go:356] "Generic (PLEG): container finished" podID="6890cb08-fe65-45db-bb41-1fee110f9a18" containerID="6634c7a094141acc717ac3e218f4ee2ce6e897394126f0ae2088ee9a5cd2c13a" exitCode=0 Sep 11 20:06:57.457412 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:57.457399 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" event={"ID":"6890cb08-fe65-45db-bb41-1fee110f9a18","Type":"ContainerDied","Data":"6634c7a094141acc717ac3e218f4ee2ce6e897394126f0ae2088ee9a5cd2c13a"} Sep 11 20:06:57.457991 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:57.457450 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" event={"ID":"6890cb08-fe65-45db-bb41-1fee110f9a18","Type":"ContainerStarted","Data":"18fce44600ac90af17cd8650e0f8532d52005b2f96c058e3de6759eda42a3c41"} Sep 11 20:06:58.075126 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:58.075092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:58.605976 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:58.605952 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" Sep 11 20:06:58.689186 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:58.689150 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6890cb08-fe65-45db-bb41-1fee110f9a18-maas-api-service-ca\") pod \"6890cb08-fe65-45db-bb41-1fee110f9a18\" (UID: \"6890cb08-fe65-45db-bb41-1fee110f9a18\") " Sep 11 20:06:58.689520 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:58.689497 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6890cb08-fe65-45db-bb41-1fee110f9a18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6890cb08-fe65-45db-bb41-1fee110f9a18" (UID: "6890cb08-fe65-45db-bb41-1fee110f9a18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:06:58.790068 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:58.790026 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6890cb08-fe65-45db-bb41-1fee110f9a18-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:06:58.873116 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:58.873066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:06:59.470498 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:59.470470 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" Sep 11 20:06:59.470498 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:59.470481 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn" event={"ID":"6890cb08-fe65-45db-bb41-1fee110f9a18","Type":"ContainerDied","Data":"18fce44600ac90af17cd8650e0f8532d52005b2f96c058e3de6759eda42a3c41"} Sep 11 20:06:59.470717 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:59.470532 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18fce44600ac90af17cd8650e0f8532d52005b2f96c058e3de6759eda42a3c41" Sep 11 20:06:59.507116 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:06:59.507081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:00.351236 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:00.351202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:06.793571 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:06.793533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:06.843913 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:06.843880 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z"] Sep 11 20:07:06.844150 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:06.844122 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" podUID="6d9ed380-e785-4bb7-9220-80950a405f70" containerName="payload-pre-processing" containerID="cri-o://e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152" gracePeriod=30 Sep 11 20:07:06.895790 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:06.895766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:06.995168 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:06.995134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:07.025776 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.025722 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn"] Sep 11 20:07:07.028758 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.028731 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2f8fd9-r2nnn"] Sep 11 20:07:07.067103 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.067025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:07.105710 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.105681 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:07:07.187853 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.172605 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9ed380-e785-4bb7-9220-80950a405f70-plugins-config-volume\") pod \"6d9ed380-e785-4bb7-9220-80950a405f70\" (UID: \"6d9ed380-e785-4bb7-9220-80950a405f70\") " Sep 11 20:07:07.187853 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.172662 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmzn7\" (UniqueName: \"kubernetes.io/projected/6d9ed380-e785-4bb7-9220-80950a405f70-kube-api-access-pmzn7\") pod \"6d9ed380-e785-4bb7-9220-80950a405f70\" (UID: \"6d9ed380-e785-4bb7-9220-80950a405f70\") " Sep 11 20:07:07.187853 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.173587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9ed380-e785-4bb7-9220-80950a405f70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d9ed380-e785-4bb7-9220-80950a405f70" (UID: "6d9ed380-e785-4bb7-9220-80950a405f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:07.187853 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.176057 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9ed380-e785-4bb7-9220-80950a405f70-kube-api-access-pmzn7" (OuterVolumeSpecName: "kube-api-access-pmzn7") pod "6d9ed380-e785-4bb7-9220-80950a405f70" (UID: "6d9ed380-e785-4bb7-9220-80950a405f70"). InnerVolumeSpecName "kube-api-access-pmzn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:07.273947 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.273906 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9ed380-e785-4bb7-9220-80950a405f70-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:07.273947 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.273940 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmzn7\" (UniqueName: \"kubernetes.io/projected/6d9ed380-e785-4bb7-9220-80950a405f70-kube-api-access-pmzn7\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:07.394237 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.394150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:07.394581 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.394560 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6890cb08-fe65-45db-bb41-1fee110f9a18" path="/var/lib/kubelet/pods/6890cb08-fe65-45db-bb41-1fee110f9a18/volumes" Sep 11 20:07:07.510048 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.510006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:07.520943 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.520911 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d9ed380-e785-4bb7-9220-80950a405f70" containerID="e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152" exitCode=0 Sep 11 20:07:07.521105 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.520961 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" Sep 11 20:07:07.521105 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.520973 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" event={"ID":"6d9ed380-e785-4bb7-9220-80950a405f70","Type":"ContainerDied","Data":"e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152"} Sep 11 20:07:07.521105 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.521018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z" event={"ID":"6d9ed380-e785-4bb7-9220-80950a405f70","Type":"ContainerDied","Data":"3b68f4ea145ea8b9b8387a64bc8f3086ce867769d3d0144243f487fd2fe1cc88"} Sep 11 20:07:07.521105 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.521040 2579 scope.go:117] "RemoveContainer" containerID="e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152" Sep 11 20:07:07.530371 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.530349 2579 scope.go:117] "RemoveContainer" containerID="e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152" Sep 11 20:07:07.530687 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:07:07.530667 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152\": container with ID starting with e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152 not found: ID does not exist" containerID="e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152" Sep 11 20:07:07.530744 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.530696 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152"} err="failed to get container status \"e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152\": rpc error: code = NotFound desc = could not find container \"e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152\": container with ID starting with e5c28b0533515cbe5077b842b4957365970ea00f4d7421406eb5e21968527152 not found: ID does not exist" Sep 11 20:07:07.543418 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.543388 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z"] Sep 11 20:07:07.545645 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:07.545624 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2f8fd9-56959f4dc4-9mp9z"] Sep 11 20:07:08.281370 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:08.281340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:08.981377 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:08.981344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:09.393723 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:09.393690 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d9ed380-e785-4bb7-9220-80950a405f70" path="/var/lib/kubelet/pods/6d9ed380-e785-4bb7-9220-80950a405f70/volumes" Sep 11 20:07:09.672499 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:09.672415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:10.304417 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:10.304385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:13.583500 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:13.583464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:13.683929 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:13.683894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:13.823042 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:13.823011 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn"] Sep 11 20:07:13.823229 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:13.823209 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" containerID="cri-o://18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7" gracePeriod=30 Sep 11 20:07:13.840771 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:13.840680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:18.027165 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:18.027132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:18.655106 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:18.655076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:19.070050 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.070026 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:07:19.196497 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-certs\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.196497 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196463 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-socket\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.196497 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196491 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-data\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196530 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-credential-socket\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196556 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-envoy\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196612 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r69f\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-kube-api-access-9r69f\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196641 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-podinfo\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196663 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315cea54-3be9-4612-9a3a-75becf2a9d11-istiod-ca-cert\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196697 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-token\") pod \"315cea54-3be9-4612-9a3a-75becf2a9d11\" (UID: \"315cea54-3be9-4612-9a3a-75becf2a9d11\") " Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196709 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196879 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-data" (OuterVolumeSpecName: "istio-data") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196902 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:07:19.197020 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.196908 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:07:19.197435 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.197177 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.197435 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.197198 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-workload-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.197435 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.197212 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-data\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.197435 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.197225 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-credential-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.197435 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.197300 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315cea54-3be9-4612-9a3a-75becf2a9d11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:19.199122 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.199099 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:07:19.199243 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.199142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:07:19.199513 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.199490 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-kube-api-access-9r69f" (OuterVolumeSpecName: "kube-api-access-9r69f") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "kube-api-access-9r69f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:19.199513 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.199499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-token" (OuterVolumeSpecName: "istio-token") pod "315cea54-3be9-4612-9a3a-75becf2a9d11" (UID: "315cea54-3be9-4612-9a3a-75becf2a9d11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:19.298055 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.298015 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-envoy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.298055 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.298048 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r69f\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-kube-api-access-9r69f\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.298055 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.298060 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-podinfo\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.298294 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.298069 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315cea54-3be9-4612-9a3a-75becf2a9d11-istiod-ca-cert\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.298294 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.298078 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315cea54-3be9-4612-9a3a-75becf2a9d11-istio-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:19.411537 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.411510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:19.590650 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.590615 2579 generic.go:356] "Generic (PLEG): container finished" podID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerID="18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7" exitCode=0 Sep 11 20:07:19.590849 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.590666 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" Sep 11 20:07:19.590849 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.590694 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" event={"ID":"315cea54-3be9-4612-9a3a-75becf2a9d11","Type":"ContainerDied","Data":"18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7"} Sep 11 20:07:19.590849 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.590736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn" event={"ID":"315cea54-3be9-4612-9a3a-75becf2a9d11","Type":"ContainerDied","Data":"b8544f8bfb8eb33bd7eaa93357799ff5aa426e62d129e0e71b8dc645757644ed"} Sep 11 20:07:19.590849 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.590757 2579 scope.go:117] "RemoveContainer" containerID="18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7" Sep 11 20:07:19.600159 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.600108 2579 scope.go:117] "RemoveContainer" containerID="1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2" Sep 11 20:07:19.609042 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.609014 2579 scope.go:117] "RemoveContainer" containerID="18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7" Sep 11 20:07:19.609459 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:07:19.609424 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7\": container with ID starting with 18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7 not found: ID does not exist" containerID="18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7" Sep 11 20:07:19.609569 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.609467 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7"} err="failed to get container status \"18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7\": rpc error: code = NotFound desc = could not find container \"18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7\": container with ID starting with 18fa90a3356f354d06b359f94af129cca1ae7e96f032eb6b12f5fa54853261d7 not found: ID does not exist" Sep 11 20:07:19.609569 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.609494 2579 scope.go:117] "RemoveContainer" containerID="1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2" Sep 11 20:07:19.610433 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:07:19.610390 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2\": container with ID starting with 1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2 not found: ID does not exist" containerID="1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2" Sep 11 20:07:19.610557 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.610432 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2"} err="failed to get container status \"1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2\": rpc error: code = NotFound desc = could not find container \"1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2\": container with ID starting with 1583b35b0277c25c00ac5a7fef5da510e6bb9d4b582eaaed7d7e360d01e91db2 not found: ID does not exist" Sep 11 20:07:19.613329 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.613306 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn"] Sep 11 20:07:19.615585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:19.615565 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2f8fd9-openshift-default-54c88f85db-mqckn"] Sep 11 20:07:20.145784 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:20.145748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:20.771451 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:20.771423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:21.394292 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:21.394256 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" path="/var/lib/kubelet/pods/315cea54-3be9-4612-9a3a-75becf2a9d11/volumes" Sep 11 20:07:32.683422 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:32.683390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:32.713490 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:32.713450 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt"] Sep 11 20:07:32.713762 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:32.713730 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" podUID="7d262392-3996-48ab-9585-3355d9b97e17" containerName="payload-processing" containerID="cri-o://26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414" gracePeriod=30 Sep 11 20:07:32.723096 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:32.723063 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g"] Sep 11 20:07:32.723525 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:32.723490 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" podUID="5bbd04b7-84b4-452b-b6e3-b05f12fd890e" containerName="payload-pre-processing" containerID="cri-o://90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df" gracePeriod=30 Sep 11 20:07:32.759858 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:32.759794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:32.831966 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:32.831936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:32.912363 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:32.912333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:33.029461 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.029436 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:07:33.032743 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.032717 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:07:33.033275 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.033258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262392-3996-48ab-9585-3355d9b97e17-plugins-config-volume\") pod \"7d262392-3996-48ab-9585-3355d9b97e17\" (UID: \"7d262392-3996-48ab-9585-3355d9b97e17\") " Sep 11 20:07:33.033340 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.033329 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5792\" (UniqueName: \"kubernetes.io/projected/7d262392-3996-48ab-9585-3355d9b97e17-kube-api-access-t5792\") pod \"7d262392-3996-48ab-9585-3355d9b97e17\" (UID: \"7d262392-3996-48ab-9585-3355d9b97e17\") " Sep 11 20:07:33.033635 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.033614 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d262392-3996-48ab-9585-3355d9b97e17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d262392-3996-48ab-9585-3355d9b97e17" (UID: "7d262392-3996-48ab-9585-3355d9b97e17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:33.035593 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.035572 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d262392-3996-48ab-9585-3355d9b97e17-kube-api-access-t5792" (OuterVolumeSpecName: "kube-api-access-t5792") pod "7d262392-3996-48ab-9585-3355d9b97e17" (UID: "7d262392-3996-48ab-9585-3355d9b97e17"). InnerVolumeSpecName "kube-api-access-t5792". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:33.104553 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.104521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:33.134242 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.134211 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv7rt\" (UniqueName: \"kubernetes.io/projected/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-kube-api-access-tv7rt\") pod \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\" (UID: \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\") " Sep 11 20:07:33.134443 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.134278 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-plugins-config-volume\") pod \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\" (UID: \"5bbd04b7-84b4-452b-b6e3-b05f12fd890e\") " Sep 11 20:07:33.134601 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.134582 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5792\" (UniqueName: \"kubernetes.io/projected/7d262392-3996-48ab-9585-3355d9b97e17-kube-api-access-t5792\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:33.134656 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.134590 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5bbd04b7-84b4-452b-b6e3-b05f12fd890e" (UID: "5bbd04b7-84b4-452b-b6e3-b05f12fd890e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:33.134656 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.134606 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262392-3996-48ab-9585-3355d9b97e17-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:33.136473 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.136448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-kube-api-access-tv7rt" (OuterVolumeSpecName: "kube-api-access-tv7rt") pod "5bbd04b7-84b4-452b-b6e3-b05f12fd890e" (UID: "5bbd04b7-84b4-452b-b6e3-b05f12fd890e"). InnerVolumeSpecName "kube-api-access-tv7rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:33.225683 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.225645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:33.236027 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.235996 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv7rt\" (UniqueName: \"kubernetes.io/projected/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-kube-api-access-tv7rt\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:33.236027 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.236022 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd04b7-84b4-452b-b6e3-b05f12fd890e-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:07:33.673420 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.673327 2579 generic.go:356] "Generic (PLEG): container finished" podID="7d262392-3996-48ab-9585-3355d9b97e17" containerID="26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414" exitCode=0 Sep 11 20:07:33.673420 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.673398 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" Sep 11 20:07:33.673621 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.673411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" event={"ID":"7d262392-3996-48ab-9585-3355d9b97e17","Type":"ContainerDied","Data":"26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414"} Sep 11 20:07:33.673621 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.673470 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt" event={"ID":"7d262392-3996-48ab-9585-3355d9b97e17","Type":"ContainerDied","Data":"01921e6355287cb5e0e89ae0284825e75b3e9843a5d883cdfcc60567aa85f715"} Sep 11 20:07:33.673621 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.673487 2579 scope.go:117] "RemoveContainer" containerID="26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414" Sep 11 20:07:33.675076 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.675043 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bbd04b7-84b4-452b-b6e3-b05f12fd890e" containerID="90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df" exitCode=0 Sep 11 20:07:33.675220 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.675093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" event={"ID":"5bbd04b7-84b4-452b-b6e3-b05f12fd890e","Type":"ContainerDied","Data":"90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df"} Sep 11 20:07:33.675220 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.675109 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" Sep 11 20:07:33.675220 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.675134 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g" event={"ID":"5bbd04b7-84b4-452b-b6e3-b05f12fd890e","Type":"ContainerDied","Data":"278f827d5060a924eea642d5f8e01ff2413355e3fc57e50214e2aff4633e3a30"} Sep 11 20:07:33.683009 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.682990 2579 scope.go:117] "RemoveContainer" containerID="26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414" Sep 11 20:07:33.683363 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:07:33.683338 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414\": container with ID starting with 26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414 not found: ID does not exist" containerID="26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414" Sep 11 20:07:33.683454 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.683379 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414"} err="failed to get container status \"26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414\": rpc error: code = NotFound desc = could not find container \"26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414\": container with ID starting with 26910d6789f4deff65c7ed9f71dd1160826d4acb4f8f04f81106ee6f8fb67414 not found: ID does not exist" Sep 11 20:07:33.683454 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.683404 2579 scope.go:117] "RemoveContainer" containerID="90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df" Sep 11 20:07:33.692042 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.692007 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt"] Sep 11 20:07:33.692775 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.692755 2579 scope.go:117] "RemoveContainer" containerID="90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df" Sep 11 20:07:33.693200 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:07:33.693178 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df\": container with ID starting with 90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df not found: ID does not exist" containerID="90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df" Sep 11 20:07:33.693262 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.693203 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df"} err="failed to get container status \"90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df\": rpc error: code = NotFound desc = could not find container \"90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df\": container with ID starting with 90fd1cf8c8be8705c7018dd47c1d0596cdace916e39fcab5ce5317bb1a49f3df not found: ID does not exist" Sep 11 20:07:33.697791 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.697765 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-96e341-758f497485-qdvmt"] Sep 11 20:07:33.713573 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.713537 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g"] Sep 11 20:07:33.718771 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.718733 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-96e341-5d678db778-xnp8g"] Sep 11 20:07:33.959228 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:33.959125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:34.777950 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:34.777915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:35.393929 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:35.393895 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bbd04b7-84b4-452b-b6e3-b05f12fd890e" path="/var/lib/kubelet/pods/5bbd04b7-84b4-452b-b6e3-b05f12fd890e/volumes" Sep 11 20:07:35.394266 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:35.394254 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d262392-3996-48ab-9585-3355d9b97e17" path="/var/lib/kubelet/pods/7d262392-3996-48ab-9585-3355d9b97e17/volumes" Sep 11 20:07:35.591451 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:35.591422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:37.053798 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:37.053749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:07:38.254054 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:07:38.254013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:14.743924 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:14.743869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:15.736807 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:15.736770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:21.455698 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:21.455606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:21.526788 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:21.526758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:21.731963 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:21.731870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:22.052003 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.051968 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr"] Sep 11 20:08:22.052457 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052441 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6890cb08-fe65-45db-bb41-1fee110f9a18" containerName="revoke-keys" Sep 11 20:08:22.052506 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052462 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6890cb08-fe65-45db-bb41-1fee110f9a18" containerName="revoke-keys" Sep 11 20:08:22.052506 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052479 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" Sep 11 20:08:22.052506 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052484 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" Sep 11 20:08:22.052506 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052490 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d9ed380-e785-4bb7-9220-80950a405f70" containerName="payload-pre-processing" Sep 11 20:08:22.052506 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052497 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d9ed380-e785-4bb7-9220-80950a405f70" containerName="payload-pre-processing" Sep 11 20:08:22.052506 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052507 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052513 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052522 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bbd04b7-84b4-452b-b6e3-b05f12fd890e" containerName="payload-pre-processing" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052529 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bbd04b7-84b4-452b-b6e3-b05f12fd890e" containerName="payload-pre-processing" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052544 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d262392-3996-48ab-9585-3355d9b97e17" containerName="payload-processing" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052550 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d262392-3996-48ab-9585-3355d9b97e17" containerName="payload-processing" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052620 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d9ed380-e785-4bb7-9220-80950a405f70" containerName="payload-pre-processing" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052630 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052636 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d262392-3996-48ab-9585-3355d9b97e17" containerName="payload-processing" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052643 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bbd04b7-84b4-452b-b6e3-b05f12fd890e" containerName="payload-pre-processing" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052649 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="315cea54-3be9-4612-9a3a-75becf2a9d11" containerName="istio-proxy" Sep 11 20:08:22.052695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.052655 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6890cb08-fe65-45db-bb41-1fee110f9a18" containerName="revoke-keys" Sep 11 20:08:22.055884 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.055867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" Sep 11 20:08:22.058583 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.058558 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 20:08:22.058730 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.058585 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 20:08:22.063324 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.063296 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr"] Sep 11 20:08:22.137859 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.137785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9225dca3-8bd1-40a4-83ab-6122906e2c52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr\" (UID: \"9225dca3-8bd1-40a4-83ab-6122906e2c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" Sep 11 20:08:22.191326 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.191274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:22.239056 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.239020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9225dca3-8bd1-40a4-83ab-6122906e2c52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr\" (UID: \"9225dca3-8bd1-40a4-83ab-6122906e2c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" Sep 11 20:08:22.239692 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.239666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9225dca3-8bd1-40a4-83ab-6122906e2c52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr\" (UID: \"9225dca3-8bd1-40a4-83ab-6122906e2c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" Sep 11 20:08:22.368439 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.368348 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" Sep 11 20:08:22.644963 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.644937 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr"] Sep 11 20:08:22.648093 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:08:22.648066 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9225dca3_8bd1_40a4_83ab_6122906e2c52.slice/crio-87c7d21144b47aa963546257578b032b8bbb20c3a7a984be25f9ec7e998ab62f WatchSource:0}: Error finding container 87c7d21144b47aa963546257578b032b8bbb20c3a7a984be25f9ec7e998ab62f: Status 404 returned error can't find the container with id 87c7d21144b47aa963546257578b032b8bbb20c3a7a984be25f9ec7e998ab62f Sep 11 20:08:22.944941 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:22.944846 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" event={"ID":"9225dca3-8bd1-40a4-83ab-6122906e2c52","Type":"ContainerStarted","Data":"87c7d21144b47aa963546257578b032b8bbb20c3a7a984be25f9ec7e998ab62f"} Sep 11 20:08:23.017816 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:23.017778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:23.951976 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:23.951945 2579 generic.go:356] "Generic (PLEG): container finished" podID="9225dca3-8bd1-40a4-83ab-6122906e2c52" containerID="ba0cf231e4aae6bb34845236fa571a49d02b579ea2195644d2b99cd30fbe709b" exitCode=0 Sep 11 20:08:23.952358 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:23.952014 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" event={"ID":"9225dca3-8bd1-40a4-83ab-6122906e2c52","Type":"ContainerDied","Data":"ba0cf231e4aae6bb34845236fa571a49d02b579ea2195644d2b99cd30fbe709b"} Sep 11 20:08:24.098536 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:24.098503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:25.031501 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.031473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:25.104361 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.104332 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" Sep 11 20:08:25.168092 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.168051 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9225dca3-8bd1-40a4-83ab-6122906e2c52-maas-api-service-ca\") pod \"9225dca3-8bd1-40a4-83ab-6122906e2c52\" (UID: \"9225dca3-8bd1-40a4-83ab-6122906e2c52\") " Sep 11 20:08:25.168450 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.168422 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9225dca3-8bd1-40a4-83ab-6122906e2c52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9225dca3-8bd1-40a4-83ab-6122906e2c52" (UID: "9225dca3-8bd1-40a4-83ab-6122906e2c52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:08:25.168548 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.168530 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9225dca3-8bd1-40a4-83ab-6122906e2c52-maas-api-service-ca\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:08:25.753974 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.753891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:25.963356 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.963312 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" event={"ID":"9225dca3-8bd1-40a4-83ab-6122906e2c52","Type":"ContainerDied","Data":"87c7d21144b47aa963546257578b032b8bbb20c3a7a984be25f9ec7e998ab62f"} Sep 11 20:08:25.963528 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.963368 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87c7d21144b47aa963546257578b032b8bbb20c3a7a984be25f9ec7e998ab62f" Sep 11 20:08:25.963528 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:25.963322 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr" Sep 11 20:08:32.659924 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:32.659881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:32.742498 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:32.742469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:32.835978 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:32.835919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:32.883171 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:32.883133 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr"] Sep 11 20:08:32.886525 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:32.886497 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df1a8a-9ssqr"] Sep 11 20:08:32.917042 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:32.916959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:33.228102 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:33.228020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:33.328278 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:33.328227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:33.393280 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:33.393247 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9225dca3-8bd1-40a4-83ab-6122906e2c52" path="/var/lib/kubelet/pods/9225dca3-8bd1-40a4-83ab-6122906e2c52/volumes" Sep 11 20:08:34.108127 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:34.108086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:34.975652 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:34.975611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:39.187953 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:39.187917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:39.289222 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:39.289186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:39.436740 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:39.436713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:39.920946 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:39.920916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:40.777569 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:40.777534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:41.467077 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:41.467039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:42.191055 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:42.191022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:50.334942 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.334901 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p"] Sep 11 20:08:50.339369 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.339340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:50.340730 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.340708 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-57d97d-29819280-xks5p"] Sep 11 20:08:50.362416 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.362362 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf"] Sep 11 20:08:50.362690 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.362656 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" podUID="919b4976-acd0-4389-bf35-e1ffdc88b287" containerName="payload-processing" containerID="cri-o://5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c" gracePeriod=30 Sep 11 20:08:50.372506 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.372475 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5"] Sep 11 20:08:50.372695 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.372671 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" podUID="fec57c7e-70a9-44f1-9f67-664f9fe2fd15" containerName="payload-pre-processing" containerID="cri-o://42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863" gracePeriod=30 Sep 11 20:08:50.410518 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.410485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:50.477204 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.477165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:50.535981 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.535955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:50.657244 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.657205 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 20:08:50.660094 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.660071 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 20:08:50.736478 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.736442 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgsnj\" (UniqueName: \"kubernetes.io/projected/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-kube-api-access-dgsnj\") pod \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\" (UID: \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\") " Sep 11 20:08:50.736661 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.736498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-plugins-config-volume\") pod \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\" (UID: \"fec57c7e-70a9-44f1-9f67-664f9fe2fd15\") " Sep 11 20:08:50.736661 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.736546 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlkxd\" (UniqueName: \"kubernetes.io/projected/919b4976-acd0-4389-bf35-e1ffdc88b287-kube-api-access-xlkxd\") pod \"919b4976-acd0-4389-bf35-e1ffdc88b287\" (UID: \"919b4976-acd0-4389-bf35-e1ffdc88b287\") " Sep 11 20:08:50.736661 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.736639 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919b4976-acd0-4389-bf35-e1ffdc88b287-plugins-config-volume\") pod \"919b4976-acd0-4389-bf35-e1ffdc88b287\" (UID: \"919b4976-acd0-4389-bf35-e1ffdc88b287\") " Sep 11 20:08:50.737104 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.737077 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fec57c7e-70a9-44f1-9f67-664f9fe2fd15" (UID: "fec57c7e-70a9-44f1-9f67-664f9fe2fd15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:08:50.737202 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.737079 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919b4976-acd0-4389-bf35-e1ffdc88b287-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "919b4976-acd0-4389-bf35-e1ffdc88b287" (UID: "919b4976-acd0-4389-bf35-e1ffdc88b287"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:08:50.738999 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.738977 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919b4976-acd0-4389-bf35-e1ffdc88b287-kube-api-access-xlkxd" (OuterVolumeSpecName: "kube-api-access-xlkxd") pod "919b4976-acd0-4389-bf35-e1ffdc88b287" (UID: "919b4976-acd0-4389-bf35-e1ffdc88b287"). InnerVolumeSpecName "kube-api-access-xlkxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:08:50.739110 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.739087 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-kube-api-access-dgsnj" (OuterVolumeSpecName: "kube-api-access-dgsnj") pod "fec57c7e-70a9-44f1-9f67-664f9fe2fd15" (UID: "fec57c7e-70a9-44f1-9f67-664f9fe2fd15"). InnerVolumeSpecName "kube-api-access-dgsnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:08:50.838303 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.838270 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919b4976-acd0-4389-bf35-e1ffdc88b287-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:08:50.838303 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.838302 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgsnj\" (UniqueName: \"kubernetes.io/projected/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-kube-api-access-dgsnj\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:08:50.838530 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.838314 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec57c7e-70a9-44f1-9f67-664f9fe2fd15-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:08:50.838530 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.838327 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlkxd\" (UniqueName: \"kubernetes.io/projected/919b4976-acd0-4389-bf35-e1ffdc88b287-kube-api-access-xlkxd\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:08:50.866429 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.866394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:50.974355 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:50.974268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:51.097436 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.097397 2579 generic.go:356] "Generic (PLEG): container finished" podID="919b4976-acd0-4389-bf35-e1ffdc88b287" containerID="5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c" exitCode=0 Sep 11 20:08:51.097624 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.097445 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" Sep 11 20:08:51.097624 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.097487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" event={"ID":"919b4976-acd0-4389-bf35-e1ffdc88b287","Type":"ContainerDied","Data":"5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c"} Sep 11 20:08:51.097624 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.097537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf" event={"ID":"919b4976-acd0-4389-bf35-e1ffdc88b287","Type":"ContainerDied","Data":"dd35c464491f02bbedacff48a07aae3585ab9b6f8d70a2390dde50b2390c3d42"} Sep 11 20:08:51.097624 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.097565 2579 scope.go:117] "RemoveContainer" containerID="5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c" Sep 11 20:08:51.099019 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.098994 2579 generic.go:356] "Generic (PLEG): container finished" podID="fec57c7e-70a9-44f1-9f67-664f9fe2fd15" containerID="42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863" exitCode=0 Sep 11 20:08:51.099145 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.099037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" event={"ID":"fec57c7e-70a9-44f1-9f67-664f9fe2fd15","Type":"ContainerDied","Data":"42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863"} Sep 11 20:08:51.099145 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.099060 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" Sep 11 20:08:51.099145 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.099071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5" event={"ID":"fec57c7e-70a9-44f1-9f67-664f9fe2fd15","Type":"ContainerDied","Data":"1156db301fdf23f65c5e17770e4966b7f7289ea65978179546d2b8ed29423fa3"} Sep 11 20:08:51.109089 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.109065 2579 scope.go:117] "RemoveContainer" containerID="5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c" Sep 11 20:08:51.109457 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:08:51.109388 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c\": container with ID starting with 5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c not found: ID does not exist" containerID="5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c" Sep 11 20:08:51.109457 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.109420 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c"} err="failed to get container status \"5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c\": rpc error: code = NotFound desc = could not find container \"5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c\": container with ID starting with 5d7384ce1011d409cabd41fd60cb4dbdb5251e71e6ca6b4dfde930034011006c not found: ID does not exist" Sep 11 20:08:51.109457 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.109445 2579 scope.go:117] "RemoveContainer" containerID="42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863" Sep 11 20:08:51.118066 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.118044 2579 scope.go:117] "RemoveContainer" containerID="42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863" Sep 11 20:08:51.118406 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:08:51.118384 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863\": container with ID starting with 42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863 not found: ID does not exist" containerID="42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863" Sep 11 20:08:51.118498 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.118413 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863"} err="failed to get container status \"42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863\": rpc error: code = NotFound desc = could not find container \"42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863\": container with ID starting with 42ea75b58e5a18f46b3fd5bf02d72582491f90d679137fa9c8d8e1fd58fb8863 not found: ID does not exist" Sep 11 20:08:51.120452 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.120431 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf"] Sep 11 20:08:51.124283 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.124256 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-57d97d-6f866b6fcf-fs8cf"] Sep 11 20:08:51.135476 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.135446 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5"] Sep 11 20:08:51.139307 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.139280 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-57d97d-67595d44c4-9wdp5"] Sep 11 20:08:51.393983 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.393952 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919b4976-acd0-4389-bf35-e1ffdc88b287" path="/var/lib/kubelet/pods/919b4976-acd0-4389-bf35-e1ffdc88b287/volumes" Sep 11 20:08:51.394372 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.394249 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e679464e-5afa-430d-9f26-d72f042dacbc" path="/var/lib/kubelet/pods/e679464e-5afa-430d-9f26-d72f042dacbc/volumes" Sep 11 20:08:51.394562 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.394545 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec57c7e-70a9-44f1-9f67-664f9fe2fd15" path="/var/lib/kubelet/pods/fec57c7e-70a9-44f1-9f67-664f9fe2fd15/volumes" Sep 11 20:08:51.580304 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:51.580269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:56.814470 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:56.814437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:56.917853 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:56.917803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:57.047809 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:57.047769 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95"] Sep 11 20:08:57.048086 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:57.048060 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" containerID="cri-o://22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b" gracePeriod=30 Sep 11 20:08:57.071464 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:57.071377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:57.594636 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:57.594603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:57.892341 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:57.892243 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 20:08:57.892341 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:57.892306 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 20:08:58.498656 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:58.498617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:08:59.149299 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:08:59.149238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:02.306561 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.306536 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 20:09:02.355525 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355444 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l598c\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-kube-api-access-l598c\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.355525 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-socket\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.355729 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355544 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istiod-ca-cert\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.355729 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355602 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-podinfo\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.355729 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355626 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-certs\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.355729 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355654 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-credential-socket\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.355729 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355680 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-data\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.356006 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355759 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-token\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.356006 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355841 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-envoy\") pod \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\" (UID: \"45eca849-ed2b-493f-8c8f-5d04ec610ee9\") " Sep 11 20:09:02.356332 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.355880 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:02.356445 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.356094 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-data" (OuterVolumeSpecName: "istio-data") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:02.356445 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.356142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:09:02.356445 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.356269 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:02.356445 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.356283 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:02.357985 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.357956 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:09:02.358136 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.358115 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-token" (OuterVolumeSpecName: "istio-token") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:09:02.358405 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.358384 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-kube-api-access-l598c" (OuterVolumeSpecName: "kube-api-access-l598c") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "kube-api-access-l598c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:09:02.358405 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.358396 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45eca849-ed2b-493f-8c8f-5d04ec610ee9" (UID: "45eca849-ed2b-493f-8c8f-5d04ec610ee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:02.456979 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.456940 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-podinfo\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:02.456979 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.456972 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:02.456979 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.456984 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-credential-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:02.457246 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.456995 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-data\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:02.457246 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.457006 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:02.457246 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.457018 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istio-envoy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:02.457246 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.457028 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l598c\" (UniqueName: \"kubernetes.io/projected/45eca849-ed2b-493f-8c8f-5d04ec610ee9-kube-api-access-l598c\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:02.457246 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.457038 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45eca849-ed2b-493f-8c8f-5d04ec610ee9-workload-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:02.457246 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:02.457049 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45eca849-ed2b-493f-8c8f-5d04ec610ee9-istiod-ca-cert\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:03.168339 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.168299 2579 generic.go:356] "Generic (PLEG): container finished" podID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerID="22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b" exitCode=0 Sep 11 20:09:03.168518 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.168348 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" Sep 11 20:09:03.168518 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.168362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" event={"ID":"45eca849-ed2b-493f-8c8f-5d04ec610ee9","Type":"ContainerDied","Data":"22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b"} Sep 11 20:09:03.168518 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.168412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95" event={"ID":"45eca849-ed2b-493f-8c8f-5d04ec610ee9","Type":"ContainerDied","Data":"f86ea23e22122c12f997f0cc185ed0c5114f5f386293db6293da42ce78d88290"} Sep 11 20:09:03.168518 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.168435 2579 scope.go:117] "RemoveContainer" containerID="22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b" Sep 11 20:09:03.178418 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.178399 2579 scope.go:117] "RemoveContainer" containerID="a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925" Sep 11 20:09:03.188960 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.188839 2579 scope.go:117] "RemoveContainer" containerID="22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b" Sep 11 20:09:03.189251 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:09:03.189223 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b\": container with ID starting with 22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b not found: ID does not exist" containerID="22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b" Sep 11 20:09:03.189404 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.189381 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b"} err="failed to get container status \"22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b\": rpc error: code = NotFound desc = could not find container \"22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b\": container with ID starting with 22c99d6478381483f8bff7b1b1de60e4caa8a02bd613edc4f407d7289a28348b not found: ID does not exist" Sep 11 20:09:03.189509 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.189489 2579 scope.go:117] "RemoveContainer" containerID="a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925" Sep 11 20:09:03.189844 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:09:03.189801 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925\": container with ID starting with a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925 not found: ID does not exist" containerID="a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925" Sep 11 20:09:03.189929 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.189861 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925"} err="failed to get container status \"a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925\": rpc error: code = NotFound desc = could not find container \"a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925\": container with ID starting with a89ba533fad926fd0055afcab2def9fa9a07da46d8762efc4fc8957d83f22925 not found: ID does not exist" Sep 11 20:09:03.191477 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.191456 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95"] Sep 11 20:09:03.196804 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.196784 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-57d97d-openshift-default-6b85b8587b-5rg95"] Sep 11 20:09:03.393986 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:03.393953 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" path="/var/lib/kubelet/pods/45eca849-ed2b-493f-8c8f-5d04ec610ee9/volumes" Sep 11 20:09:07.998930 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:07.998891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:08.031928 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.031889 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb"] Sep 11 20:09:08.032400 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.032366 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" podUID="64a49f85-9a6c-4cc9-b629-4cbf83297d39" containerName="payload-pre-processing" containerID="cri-o://86dead006f0390a934614ff6977936374175d749188ff7fa40a93ebc525efa97" gracePeriod=30 Sep 11 20:09:08.068100 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.068071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:08.079396 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.079354 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" podUID="64a49f85-9a6c-4cc9-b629-4cbf83297d39" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.67:9004: connect: connection refused" Sep 11 20:09:08.168939 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.168885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:08.198509 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.198479 2579 generic.go:356] "Generic (PLEG): container finished" podID="64a49f85-9a6c-4cc9-b629-4cbf83297d39" containerID="86dead006f0390a934614ff6977936374175d749188ff7fa40a93ebc525efa97" exitCode=0 Sep 11 20:09:08.198677 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.198538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" event={"ID":"64a49f85-9a6c-4cc9-b629-4cbf83297d39","Type":"ContainerDied","Data":"86dead006f0390a934614ff6977936374175d749188ff7fa40a93ebc525efa97"} Sep 11 20:09:08.234024 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.233974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:08.294812 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.294787 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 20:09:08.419941 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.419898 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk47g\" (UniqueName: \"kubernetes.io/projected/64a49f85-9a6c-4cc9-b629-4cbf83297d39-kube-api-access-bk47g\") pod \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\" (UID: \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\") " Sep 11 20:09:08.420161 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.419971 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a49f85-9a6c-4cc9-b629-4cbf83297d39-plugins-config-volume\") pod \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\" (UID: \"64a49f85-9a6c-4cc9-b629-4cbf83297d39\") " Sep 11 20:09:08.420315 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.420291 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a49f85-9a6c-4cc9-b629-4cbf83297d39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64a49f85-9a6c-4cc9-b629-4cbf83297d39" (UID: "64a49f85-9a6c-4cc9-b629-4cbf83297d39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:09:08.422221 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.422198 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a49f85-9a6c-4cc9-b629-4cbf83297d39-kube-api-access-bk47g" (OuterVolumeSpecName: "kube-api-access-bk47g") pod "64a49f85-9a6c-4cc9-b629-4cbf83297d39" (UID: "64a49f85-9a6c-4cc9-b629-4cbf83297d39"). InnerVolumeSpecName "kube-api-access-bk47g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:09:08.520983 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.520931 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk47g\" (UniqueName: \"kubernetes.io/projected/64a49f85-9a6c-4cc9-b629-4cbf83297d39-kube-api-access-bk47g\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:08.520983 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.520969 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a49f85-9a6c-4cc9-b629-4cbf83297d39-plugins-config-volume\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:08.576642 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.576597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:08.673674 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:08.673583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:09.204893 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:09.204856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" event={"ID":"64a49f85-9a6c-4cc9-b629-4cbf83297d39","Type":"ContainerDied","Data":"7fc3663af86f152f48e829297d0a1fd2edb2e7019ec98e85c9ddcfb73d6e1ba5"} Sep 11 20:09:09.205332 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:09.204911 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb" Sep 11 20:09:09.205332 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:09.204952 2579 scope.go:117] "RemoveContainer" containerID="86dead006f0390a934614ff6977936374175d749188ff7fa40a93ebc525efa97" Sep 11 20:09:09.228102 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:09.228068 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb"] Sep 11 20:09:09.231551 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:09.231524 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8a4eb7-74fb5ff5cd-gl4tb"] Sep 11 20:09:09.393763 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:09.393660 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64a49f85-9a6c-4cc9-b629-4cbf83297d39" path="/var/lib/kubelet/pods/64a49f85-9a6c-4cc9-b629-4cbf83297d39/volumes" Sep 11 20:09:15.079540 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:15.079504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:15.302646 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:15.302611 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w"] Sep 11 20:09:15.302910 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:15.302868 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" containerID="cri-o://ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249" gracePeriod=30 Sep 11 20:09:16.205262 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:16.205228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:09:20.550242 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.550209 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 20:09:20.643220 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643120 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-data\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643220 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643179 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j2zk\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-kube-api-access-9j2zk\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643220 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643222 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-socket\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643259 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f873fa-6643-4570-94e7-c121820936c5-istiod-ca-cert\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643281 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f873fa-6643-4570-94e7-c121820936c5-istio-podinfo\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643327 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-credential-socket\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643376 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-certs\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643404 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-envoy\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643448 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-istio-token\") pod \"13f873fa-6643-4570-94e7-c121820936c5\" (UID: \"13f873fa-6643-4570-94e7-c121820936c5\") " Sep 11 20:09:20.643519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643451 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:20.643887 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643536 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:20.643887 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643648 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:20.643887 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643805 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:20.643887 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643844 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-credential-socket\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:20.643887 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643857 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-data\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:20.643887 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f873fa-6643-4570-94e7-c121820936c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:09:20.643887 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.643878 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:20.646289 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.646262 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-kube-api-access-9j2zk" (OuterVolumeSpecName: "kube-api-access-9j2zk") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "kube-api-access-9j2zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:09:20.646484 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.646461 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:09:20.646862 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.646808 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13f873fa-6643-4570-94e7-c121820936c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 20:09:20.646981 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.646883 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "13f873fa-6643-4570-94e7-c121820936c5" (UID: "13f873fa-6643-4570-94e7-c121820936c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:09:20.744773 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.744736 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-workload-certs\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:20.744773 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.744767 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f873fa-6643-4570-94e7-c121820936c5-istio-envoy\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:20.744773 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.744775 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-istio-token\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:20.744773 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.744783 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j2zk\" (UniqueName: \"kubernetes.io/projected/13f873fa-6643-4570-94e7-c121820936c5-kube-api-access-9j2zk\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:20.745076 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.744792 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f873fa-6643-4570-94e7-c121820936c5-istiod-ca-cert\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:20.745076 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:20.744800 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f873fa-6643-4570-94e7-c121820936c5-istio-podinfo\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\"" Sep 11 20:09:21.271486 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.271454 2579 generic.go:356] "Generic (PLEG): container finished" podID="13f873fa-6643-4570-94e7-c121820936c5" containerID="ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249" exitCode=0 Sep 11 20:09:21.271669 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.271503 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" Sep 11 20:09:21.271669 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.271510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" event={"ID":"13f873fa-6643-4570-94e7-c121820936c5","Type":"ContainerDied","Data":"ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249"} Sep 11 20:09:21.271669 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.271561 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w" event={"ID":"13f873fa-6643-4570-94e7-c121820936c5","Type":"ContainerDied","Data":"8808f43f900c3ee78f70948429ee19d791bfc627baf6799179ab0dcf211dc162"} Sep 11 20:09:21.271669 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.271586 2579 scope.go:117] "RemoveContainer" containerID="ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249" Sep 11 20:09:21.281229 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.281210 2579 scope.go:117] "RemoveContainer" containerID="b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13" Sep 11 20:09:21.289605 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.289589 2579 scope.go:117] "RemoveContainer" containerID="ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249" Sep 11 20:09:21.289970 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:09:21.289946 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249\": container with ID starting with ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249 not found: ID does not exist" containerID="ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249" Sep 11 20:09:21.290063 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.289981 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249"} err="failed to get container status \"ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249\": rpc error: code = NotFound desc = could not find container \"ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249\": container with ID starting with ee6e03be46ede892bbb76c89b7a56c63baa08a7fcd8b5eaf009917e3495a9249 not found: ID does not exist" Sep 11 20:09:21.290063 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.290000 2579 scope.go:117] "RemoveContainer" containerID="b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13" Sep 11 20:09:21.290272 ip-10-0-138-82 kubenswrapper[2579]: E0911 20:09:21.290257 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13\": container with ID starting with b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13 not found: ID does not exist" containerID="b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13" Sep 11 20:09:21.290324 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.290276 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13"} err="failed to get container status \"b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13\": rpc error: code = NotFound desc = could not find container \"b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13\": container with ID starting with b0c791570985171dc28035cb95a308de2b2d44dbc0bbee02a945d36fd4367c13 not found: ID does not exist" Sep 11 20:09:21.294527 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.294505 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w"] Sep 11 20:09:21.298382 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.298360 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8a4eb7-openshift-default-7cf4584cb-pth8w"] Sep 11 20:09:21.393501 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:21.393468 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f873fa-6643-4570-94e7-c121820936c5" path="/var/lib/kubelet/pods/13f873fa-6643-4570-94e7-c121820936c5/volumes" Sep 11 20:09:27.546854 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:27.546803 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 20:09:27.551125 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:27.551104 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 20:09:27.824469 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:27.824386 2579 scope.go:117] "RemoveContainer" containerID="1a7232b56683eb742197dd3401b066b916305d453b54b1992a8299f480f4517e" Sep 11 20:09:44.850439 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:09:44.850410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:04.364627 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:10:04.364547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:34.683405 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:10:34.683370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:34.772972 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:10:34.772936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:39.325872 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:10:39.325757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:10:41.073182 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:10:41.073136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:11:35.905321 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:11:35.905212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:05.391011 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:12:05.390981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:12:05.449376 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:12:05.449337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:23.197231 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:13:23.197202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:25.507248 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:13:25.507208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:13:27.884746 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:13:27.884708 2579 scope.go:117] "RemoveContainer" containerID="6634c7a094141acc717ac3e218f4ee2ce6e897394126f0ae2088ee9a5cd2c13a" Sep 11 20:13:42.294988 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:13:42.294944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:01.820548 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:14:01.820493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:03.972940 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:14:03.972911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:14:27.584496 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:14:27.584423 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 20:14:27.589034 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:14:27.589013 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7ws8_fb818bad-1ca1-468c-a001-711138c499ef/ovn-acl-logging/0.log" Sep 11 20:14:27.904398 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:14:27.904304 2579 scope.go:117] "RemoveContainer" containerID="ba0cf231e4aae6bb34845236fa571a49d02b579ea2195644d2b99cd30fbe709b" Sep 11 20:15:00.135774 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.135737 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65"] Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136180 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136197 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136208 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136213 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136226 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136233 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136249 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9225dca3-8bd1-40a4-83ab-6122906e2c52" containerName="revoke-keys" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136255 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9225dca3-8bd1-40a4-83ab-6122906e2c52" containerName="revoke-keys" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136270 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a49f85-9a6c-4cc9-b629-4cbf83297d39" containerName="payload-pre-processing" Sep 11 20:15:00.136276 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136276 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a49f85-9a6c-4cc9-b629-4cbf83297d39" containerName="payload-pre-processing" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136285 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919b4976-acd0-4389-bf35-e1ffdc88b287" containerName="payload-processing" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136290 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="919b4976-acd0-4389-bf35-e1ffdc88b287" containerName="payload-processing" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136299 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec57c7e-70a9-44f1-9f67-664f9fe2fd15" containerName="payload-pre-processing" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136304 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec57c7e-70a9-44f1-9f67-664f9fe2fd15" containerName="payload-pre-processing" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136365 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136373 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136380 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec57c7e-70a9-44f1-9f67-664f9fe2fd15" containerName="payload-pre-processing" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136386 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136393 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45eca849-ed2b-493f-8c8f-5d04ec610ee9" containerName="istio-proxy" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136399 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9225dca3-8bd1-40a4-83ab-6122906e2c52" containerName="revoke-keys" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136404 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="919b4976-acd0-4389-bf35-e1ffdc88b287" containerName="payload-processing" Sep 11 20:15:00.136585 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.136410 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a49f85-9a6c-4cc9-b629-4cbf83297d39" containerName="payload-pre-processing" Sep 11 20:15:00.139671 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.139653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" Sep 11 20:15:00.142783 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.142765 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\"" Sep 11 20:15:00.149560 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.149532 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65"] Sep 11 20:15:00.150357 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.150334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" Sep 11 20:15:00.428282 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.428257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65"] Sep 11 20:15:00.430898 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:15:00.430864 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2be99a5_dce2_48f9_99fb_16efbc6b27f9.slice/crio-4198e524c978c5c126e7072cc9a5cac98f3456777f246964bf2081e9d9d86944 WatchSource:0}: Error finding container 4198e524c978c5c126e7072cc9a5cac98f3456777f246964bf2081e9d9d86944: Status 404 returned error can't find the container with id 4198e524c978c5c126e7072cc9a5cac98f3456777f246964bf2081e9d9d86944 Sep 11 20:15:00.994615 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.994577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" event={"ID":"e2be99a5-dce2-48f9-99fb-16efbc6b27f9","Type":"ContainerStarted","Data":"6f632e5a818016ac1819e1cf2c39420b85876f17ddf74aaf0a25564f36a0ec7f"} Sep 11 20:15:00.994847 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:00.994647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" event={"ID":"e2be99a5-dce2-48f9-99fb-16efbc6b27f9","Type":"ContainerStarted","Data":"4198e524c978c5c126e7072cc9a5cac98f3456777f246964bf2081e9d9d86944"} Sep 11 20:15:01.010157 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:01.010099 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" podStartSLOduration=1.010081267 podStartE2EDuration="1.010081267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:15:01.008442118 +0000 UTC m=+2734.141754709" watchObservedRunningTime="2026-09-11 20:15:01.010081267 +0000 UTC m=+2734.143393855" Sep 11 20:15:01.186236 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:01.186206 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sl7f4_1b238d23-1b8b-4958-b502-9a6a91c8be9a/server/0.log" Sep 11 20:15:01.324230 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:01.324199 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hqd7z_c4d63cf1-3033-4801-a4ac-31b25b4b3eca/manager/0.log" Sep 11 20:15:01.439233 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:01.439150 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cccd5967-755nm_3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4/copy-manifests/0.log" Sep 11 20:15:01.446055 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:01.446035 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cccd5967-755nm_3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4/manager/0.log" Sep 11 20:15:02.000291 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:02.000243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:02.001418 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:02.001394 2579 generic.go:356] "Generic (PLEG): container finished" podID="e2be99a5-dce2-48f9-99fb-16efbc6b27f9" containerID="6f632e5a818016ac1819e1cf2c39420b85876f17ddf74aaf0a25564f36a0ec7f" exitCode=0 Sep 11 20:15:02.001502 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:02.001480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" event={"ID":"e2be99a5-dce2-48f9-99fb-16efbc6b27f9","Type":"ContainerDied","Data":"6f632e5a818016ac1819e1cf2c39420b85876f17ddf74aaf0a25564f36a0ec7f"} Sep 11 20:15:02.334040 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:02.334012 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819280-d67pj_70f21ccb-3d71-46bf-b4a5-abe059193890/cleanup/0.log" Sep 11 20:15:02.446348 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:02.446319 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819295-rhv65_e2be99a5-dce2-48f9-99fb-16efbc6b27f9/cleanup/0.log" Sep 11 20:15:03.145473 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:03.145447 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" Sep 11 20:15:03.364359 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:03.364319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:04.012578 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:04.012538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" event={"ID":"e2be99a5-dce2-48f9-99fb-16efbc6b27f9","Type":"ContainerDied","Data":"4198e524c978c5c126e7072cc9a5cac98f3456777f246964bf2081e9d9d86944"} Sep 11 20:15:04.012785 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:04.012605 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4198e524c978c5c126e7072cc9a5cac98f3456777f246964bf2081e9d9d86944" Sep 11 20:15:04.012785 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:04.012550 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819295-rhv65" Sep 11 20:15:04.318886 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:04.318856 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2_6dc26d31-389a-4c3f-856d-38c53f4d42dc/istio-proxy/0.log" Sep 11 20:15:04.433539 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:04.433509 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lxt9k_c8adb287-02d5-487c-8a6b-7e23bc2af729/discovery/0.log" Sep 11 20:15:04.750157 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:04.750076 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fvg7z_bf996906-b8e6-4323-af98-f888584e2d1c/istio-proxy/1.log" Sep 11 20:15:04.969197 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:04.969163 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2ht6_9d9a0f88-e534-415a-bfea-a3e605edc458/payload-processing/0.log" Sep 11 20:15:05.624470 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:05.624440 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-mczrs_e00b7b77-1dc2-4149-b9da-70d9e93b9cd0/main/0.log" Sep 11 20:15:05.831939 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:05.831910 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxgvwh_fc8b552c-b7ab-4bef-873c-418aaae81e3b/main/0.log" Sep 11 20:15:11.409927 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:11.409873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:11.535371 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:11.535337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" limit=1024 Sep 11 20:15:14.236933 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:14.236897 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2rwkf_c1783842-80b8-4d47-be48-86b24985f631/global-pull-secret-syncer/0.log" Sep 11 20:15:14.364431 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:14.364400 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-24lzg_52905ff5-215f-40aa-bb52-9e786b733946/konnectivity-agent/0.log" Sep 11 20:15:14.483401 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:14.483369 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-82.ec2.internal_110b986e19c57c6926c4a3f9fb96dca0/haproxy/0.log" Sep 11 20:15:21.072691 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.072663 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2f9fl_b426d294-f15a-4131-a1e9-aac7f1d583ee/kube-state-metrics/0.log" Sep 11 20:15:21.097529 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.097494 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2f9fl_b426d294-f15a-4131-a1e9-aac7f1d583ee/kube-rbac-proxy-main/0.log" Sep 11 20:15:21.118096 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.118063 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2f9fl_b426d294-f15a-4131-a1e9-aac7f1d583ee/kube-rbac-proxy-self/0.log" Sep 11 20:15:21.148226 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.148195 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5dbcf4575-fctvc_323b7006-9597-4d99-8ce5-148dd3318201/metrics-server/0.log" Sep 11 20:15:21.210940 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.210906 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78drv_1b61bd66-ef41-47dc-a16e-007810f7d3c3/node-exporter/0.log" Sep 11 20:15:21.244043 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.244018 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78drv_1b61bd66-ef41-47dc-a16e-007810f7d3c3/kube-rbac-proxy/0.log" Sep 11 20:15:21.273358 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.273325 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78drv_1b61bd66-ef41-47dc-a16e-007810f7d3c3/init-textfile/0.log" Sep 11 20:15:21.490250 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.490220 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-z7r2t_46c1d939-adf7-440c-ab3a-91544f68a055/kube-rbac-proxy-main/0.log" Sep 11 20:15:21.523134 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.523098 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-z7r2t_46c1d939-adf7-440c-ab3a-91544f68a055/kube-rbac-proxy-self/0.log" Sep 11 20:15:21.561149 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.561118 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-z7r2t_46c1d939-adf7-440c-ab3a-91544f68a055/openshift-state-metrics/0.log" Sep 11 20:15:21.643295 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.643262 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddcaf05d-a1e5-4275-b2a5-d231fdabca9f/prometheus/0.log" Sep 11 20:15:21.690463 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.690437 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddcaf05d-a1e5-4275-b2a5-d231fdabca9f/config-reloader/0.log" Sep 11 20:15:21.749632 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.749551 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddcaf05d-a1e5-4275-b2a5-d231fdabca9f/thanos-sidecar/0.log" Sep 11 20:15:21.819687 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.819655 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddcaf05d-a1e5-4275-b2a5-d231fdabca9f/kube-rbac-proxy-web/0.log" Sep 11 20:15:21.887461 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.887434 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddcaf05d-a1e5-4275-b2a5-d231fdabca9f/kube-rbac-proxy/0.log" Sep 11 20:15:21.951375 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.951350 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddcaf05d-a1e5-4275-b2a5-d231fdabca9f/kube-rbac-proxy-thanos/0.log" Sep 11 20:15:21.987375 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:21.987345 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddcaf05d-a1e5-4275-b2a5-d231fdabca9f/init-config-reloader/0.log" Sep 11 20:15:22.029160 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:22.029076 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-f2bkm_22885ef4-d0ba-4dfa-b293-b128087f0779/prometheus-operator/0.log" Sep 11 20:15:22.057643 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:22.057615 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-f2bkm_22885ef4-d0ba-4dfa-b293-b128087f0779/kube-rbac-proxy/0.log" Sep 11 20:15:22.090545 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:22.090516 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-h9z9h_db589459-af0e-43dd-882d-d0d20d4ea99a/prometheus-operator-admission-webhook/0.log" Sep 11 20:15:23.643866 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:23.643793 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-dkt69_a7e76cf0-5a58-4409-be0e-e2bcbea840ba/networking-console-plugin/0.log" Sep 11 20:15:24.659066 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.659029 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl"] Sep 11 20:15:24.659511 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.659409 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" Sep 11 20:15:24.659511 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.659420 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f873fa-6643-4570-94e7-c121820936c5" containerName="istio-proxy" Sep 11 20:15:24.659511 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.659440 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2be99a5-dce2-48f9-99fb-16efbc6b27f9" containerName="cleanup" Sep 11 20:15:24.659511 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.659446 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2be99a5-dce2-48f9-99fb-16efbc6b27f9" containerName="cleanup" Sep 11 20:15:24.659511 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.659510 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2be99a5-dce2-48f9-99fb-16efbc6b27f9" containerName="cleanup" Sep 11 20:15:24.663005 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.662985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.665939 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.665916 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89mbl\"/\"kube-root-ca.crt\"" Sep 11 20:15:24.666038 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.665988 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89mbl\"/\"openshift-service-ca.crt\"" Sep 11 20:15:24.667113 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.667096 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-89mbl\"/\"default-dockercfg-xmf8z\"" Sep 11 20:15:24.673946 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.673917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl"] Sep 11 20:15:24.749519 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.749470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-lib-modules\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.749739 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.749584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-sys\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.749739 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.749641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-podres\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.749739 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.749705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdrg\" (UniqueName: \"kubernetes.io/projected/0efb029f-8706-4862-a951-b8840699f7ff-kube-api-access-zpdrg\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.749921 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.749743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-proc\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.850551 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-podres\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.850764 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdrg\" (UniqueName: \"kubernetes.io/projected/0efb029f-8706-4862-a951-b8840699f7ff-kube-api-access-zpdrg\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.850764 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-proc\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.850764 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-lib-modules\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.850764 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-sys\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.850764 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-podres\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.850764 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-proc\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.851115 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-sys\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.851115 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.850799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0efb029f-8706-4862-a951-b8840699f7ff-lib-modules\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.859087 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.859057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdrg\" (UniqueName: \"kubernetes.io/projected/0efb029f-8706-4862-a951-b8840699f7ff-kube-api-access-zpdrg\") pod \"perf-node-gather-daemonset-jwzgl\" (UID: \"0efb029f-8706-4862-a951-b8840699f7ff\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:24.975280 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:24.975186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:25.249853 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:25.249807 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl"] Sep 11 20:15:25.253049 ip-10-0-138-82 kubenswrapper[2579]: W0911 20:15:25.253024 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0efb029f_8706_4862_a951_b8840699f7ff.slice/crio-154089378fd2f7492ba071dddaf066967df53d8e66cb57ab81bc594d1e452bcf WatchSource:0}: Error finding container 154089378fd2f7492ba071dddaf066967df53d8e66cb57ab81bc594d1e452bcf: Status 404 returned error can't find the container with id 154089378fd2f7492ba071dddaf066967df53d8e66cb57ab81bc594d1e452bcf Sep 11 20:15:26.131121 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:26.131084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" event={"ID":"0efb029f-8706-4862-a951-b8840699f7ff","Type":"ContainerStarted","Data":"88c196e6cf707c3e84668045baccf7c78e02df9048855552806416bc4f78d86d"} Sep 11 20:15:26.131551 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:26.131142 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" event={"ID":"0efb029f-8706-4862-a951-b8840699f7ff","Type":"ContainerStarted","Data":"154089378fd2f7492ba071dddaf066967df53d8e66cb57ab81bc594d1e452bcf"} Sep 11 20:15:26.131551 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:26.131274 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:26.147707 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:26.147652 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" podStartSLOduration=2.147636305 podStartE2EDuration="2.147636305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:15:26.146204217 +0000 UTC m=+2759.279516806" watchObservedRunningTime="2026-09-11 20:15:26.147636305 +0000 UTC m=+2759.280948894" Sep 11 20:15:26.787801 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:26.787768 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n7scq_d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3/dns/0.log" Sep 11 20:15:26.807006 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:26.806971 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n7scq_d4d9a8c5-3c75-47dd-b3ba-69d00b6397e3/kube-rbac-proxy/0.log" Sep 11 20:15:26.848660 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:26.848632 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w4bpk_dcffbee5-796f-4f5b-9a7d-0e28a4c065c2/dns-node-resolver/0.log" Sep 11 20:15:27.525010 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:27.524978 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mtcmq_47ffd275-74af-4c08-9970-85bdc7f3cb07/node-ca/0.log" Sep 11 20:15:28.571645 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:28.571615 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2_6dc26d31-389a-4c3f-856d-38c53f4d42dc/istio-proxy/0.log" Sep 11 20:15:29.013056 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:29.012958 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lxt9k_c8adb287-02d5-487c-8a6b-7e23bc2af729/discovery/0.log" Sep 11 20:15:29.094967 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:29.094939 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fvg7z_bf996906-b8e6-4323-af98-f888584e2d1c/istio-proxy/0.log" Sep 11 20:15:29.097798 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:29.097778 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fvg7z_bf996906-b8e6-4323-af98-f888584e2d1c/istio-proxy/1.log" Sep 11 20:15:29.166052 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:29.166003 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2ht6_9d9a0f88-e534-415a-bfea-a3e605edc458/payload-processing/0.log" Sep 11 20:15:30.064090 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:30.064054 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k5nzx_939f0c9d-6cab-4c64-adcf-8f3052559d25/serve-healthcheck-canary/0.log" Sep 11 20:15:30.582671 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:30.582633 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-6g9vp_416bd977-648b-46f7-9f45-db0b63a02ac8/insights-operator/0.log" Sep 11 20:15:30.583528 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:30.583497 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-6g9vp_416bd977-648b-46f7-9f45-db0b63a02ac8/insights-operator/1.log" Sep 11 20:15:30.611983 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:30.611952 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2gbp_50fa5062-1652-4a9c-b9ae-ee712425592e/kube-rbac-proxy/0.log" Sep 11 20:15:30.630963 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:30.630934 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2gbp_50fa5062-1652-4a9c-b9ae-ee712425592e/exporter/0.log" Sep 11 20:15:30.650679 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:30.650633 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2gbp_50fa5062-1652-4a9c-b9ae-ee712425592e/extractor/0.log" Sep 11 20:15:32.146622 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:32.146595 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-jwzgl" Sep 11 20:15:33.507747 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:33.507654 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sl7f4_1b238d23-1b8b-4958-b502-9a6a91c8be9a/server/0.log" Sep 11 20:15:33.595397 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:33.595366 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hqd7z_c4d63cf1-3033-4801-a4ac-31b25b4b3eca/manager/0.log" Sep 11 20:15:33.623567 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:33.623535 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cccd5967-755nm_3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4/manager/0.log" Sep 11 20:15:33.647426 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:33.647405 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cccd5967-755nm_3dd2d22f-71ef-48d0-83ef-dd6ca3a2d3e4/copy-manifests/0.log" Sep 11 20:15:35.000022 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:34.999992 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2_6dc26d31-389a-4c3f-856d-38c53f4d42dc/istio-proxy/0.log" Sep 11 20:15:35.277910 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:35.277796 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f6d84646c-xdcdh_d940027e-7cb3-4d02-9b6c-bf0a111557ac/manager/0.log" Sep 11 20:15:35.433367 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:35.433330 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lxt9k_c8adb287-02d5-487c-8a6b-7e23bc2af729/discovery/0.log" Sep 11 20:15:35.478739 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:35.478702 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fvg7z_bf996906-b8e6-4323-af98-f888584e2d1c/istio-proxy/0.log" Sep 11 20:15:35.481514 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:35.481494 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fvg7z_bf996906-b8e6-4323-af98-f888584e2d1c/istio-proxy/1.log" Sep 11 20:15:35.530499 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:35.530393 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2ht6_9d9a0f88-e534-415a-bfea-a3e605edc458/payload-processing/0.log" Sep 11 20:15:37.087771 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:37.087725 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-thfd2_6dc26d31-389a-4c3f-856d-38c53f4d42dc/istio-proxy/0.log" Sep 11 20:15:37.534773 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:37.534724 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lxt9k_c8adb287-02d5-487c-8a6b-7e23bc2af729/discovery/0.log" Sep 11 20:15:37.607148 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:37.607109 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fvg7z_bf996906-b8e6-4323-af98-f888584e2d1c/istio-proxy/0.log" Sep 11 20:15:37.610918 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:37.610893 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fvg7z_bf996906-b8e6-4323-af98-f888584e2d1c/istio-proxy/1.log" Sep 11 20:15:37.654350 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:37.654316 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2ht6_9d9a0f88-e534-415a-bfea-a3e605edc458/payload-processing/0.log" Sep 11 20:15:41.580443 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:41.580393 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-25zds_4b01d3d6-8e34-4571-a00f-bb2579d8a632/kube-multus/0.log" Sep 11 20:15:41.717347 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:41.717322 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lvxpr_d380b676-3396-49b0-a514-fffbbad37502/kube-multus-additional-cni-plugins/0.log" Sep 11 20:15:41.737468 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:41.737442 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lvxpr_d380b676-3396-49b0-a514-fffbbad37502/egress-router-binary-copy/0.log" Sep 11 20:15:41.756691 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:41.756662 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lvxpr_d380b676-3396-49b0-a514-fffbbad37502/cni-plugins/0.log" Sep 11 20:15:41.776439 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:41.776411 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lvxpr_d380b676-3396-49b0-a514-fffbbad37502/bond-cni-plugin/0.log" Sep 11 20:15:41.796404 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:41.796378 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lvxpr_d380b676-3396-49b0-a514-fffbbad37502/routeoverride-cni/0.log" Sep 11 20:15:41.815274 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:41.815248 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lvxpr_d380b676-3396-49b0-a514-fffbbad37502/whereabouts-cni-bincopy/0.log" Sep 11 20:15:41.833713 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:41.833646 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lvxpr_d380b676-3396-49b0-a514-fffbbad37502/whereabouts-cni/0.log" Sep 11 20:15:42.174477 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:42.174371 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k6shn_dcd00465-6964-4cd1-8dd5-fda2a726811a/network-metrics-daemon/0.log" Sep 11 20:15:42.191298 ip-10-0-138-82 kubenswrapper[2579]: I0911 20:15:42.191268 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k6shn_dcd00465-6964-4cd1-8dd5-fda2a726811a/kube-rbac-proxy/0.log"